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The purpose of this study was to use a nonexperimental, quantitative design to 
compare mail and web surveys with survey mode self-selection at two data collection 
waves. Research questions examined differences and predictabilities among 
demographics (gender, ethnicity, age, and professional employment) and response 
quality (pronoun use, item nonresponse, response extremity, yea-saying, item 
completion errors, response length, response equivalency, anecdotal comments, and 
multiple response use) by survey mode and response wave. Analyses were conducted 
using chi-squares, ANOVAs, /-tests, and binary logistic regressions. 

A questionnaire in mail and web formats containing 48 forced-choice and 
open-ended items was administered to a nonrandom sample of Illinois public school 
guidance counselors (n = 2,880). After four reminders, the adjusted response rate was 
30.56% (n = 880); 64.32% ( n = 566) by mail and 35.68% (n = 314) by web; 77.73% 
(n = 684) during wave one and 22.27% (n = 196) during wave two. Respondents were 
75% female, 86% White with a mean age of 48 years and a mean of 19 years of 
professional employment. 




Results revealed that mail respondents were older and had more years of 
professional employment than web respondents, item nonresponse was greater in web 
than in mail surveys, and response length was greater in web than in mail surveys at 
wave one. Age, response length, gender, and yea-saying had significant partial effects 
in predicting the mail survey mode. Regarding response wave, demographics and 
response quality variables were neither different nor predictive. 

Findings suggest that researchers need to consider the potential effects of 
demographic distributions in the target population when designing mail and web 
surveys. Mail and web surveys must also be carefully constructed to overcome 
potential response quality differences while maximizing the advantages of each. The 
low overall adjusted response rate and nonrandomized design limit generalizability to 
the larger group of all school counselors in the population. This study, however, 
provides practical and timely insight regarding the use of mail and web surveys with 
mode self-selection among those who responded, and offers much potential for future 



research. 




ACKNOWLEDGMENTS 



Ed first like to express my appreciation to the entities funding and/or sponsoring 
this research: the U.S. Department of Education Office of Special Education Programs 
(grant #H324C030014), the National Science Foundation (grant #0306092), Western 
Michigan University’s Career Connections Research Center (CCRC), the Illinois State 
Board of Education, and the Illinois public school guidance counselors who participated 
in this study. I’m grateful to my advisor and dissertation committee chair, Dr. Brooks 
Applegate, for his invaluable support, patience, assistance, and feedback throughout my 
doctoral work, and for challenging me to do more than I thought I could. Thank you to 
Dr. Paula Kohler for the use of her dataset, as well as for her feedback, assistance, and 
support. I’d like to recognize Dr. David Hartmann for his valuable input and comments. 
Thank you also to the CCRC staff for their friendship and encouragement. My deepest 
gratitude is to my family and close friends for their understanding and support throughout 
this process, despite seeing little of me over the last few years. Most of all, I want to 
express my forever appreciation to my parents, Dee and Joel, for always believing in me. 

Dawn M. Mackety 



n 




TABLE OF CONTENTS 



ACKNOWLEDGMENTS ii 

LIST OF TABLES viii 

LIST OF FIGURES x 

CHAPTER 

I. INTRODUCTION 1 

Statement of the Problem 1 

Background 3 

Theoretical Framework 6 

Demographics 6 

Response Quality 8 

Early and Late Responders 13 

Research Questions 14 

Significance of the Research 16 

Delimitations 18 

Definitions 20 

Organization of the Dissertation 27 

II. LITERATURE REVIEW 29 

Brief History of Survey Research 29 

Survey Research Characteristics 31 

Questionnaire Design 32 

Costs 33 



iii 



Incentives 



35 




Table of Contents — Continued 



CHAPTER 

Response Rate 36 

Early and Late Responders 42 

Survey Research Modes 45 

Self-Administered Versus Interview Surveys 45 

Survey Mode Effects 47 

Mixed-Mode Mail and Web Surveys 49 

Respondent Demographics 49 

Gender 50 

Ethnicity 52 

Age 52 

Professional Experience 53 

Response Quality 53 

Pronoun Use 55 

Item Nonresponse 55 

Item Completion Error 58 

Response Length 59 

Yea-Saying 60 

Response Extremity 61 

Response Equivalency 62 

Multiple Response Use and Anecdotal Comments 63 

III. METHODOLOGY 66 

Research Design 66 



IV 




Table of Contents — Continued 



CHAPTER 

Using an Existing Dataset 68 

Research Questions 71 

Variables 72 

Instrumentation 73 

Content, Validity, and Reliability 73 

Item Construction 76 

Survey Modes 78 

Field Testing 84 

Data Collection 84 

Population and Sampling 87 

Statistical Analysis 88 

Chi-square (% 2 ) Analysis 88 

Analysis of Variance (ANOV A) 90 

/-test Analysis 95 

Logistic Regression Analysis 96 

Analysis of Research Questions 99 

Research Question One 99 

Research Question Two 100 

Research Question Three 112 

IV. RESULTS 114 

Response Rates 1 14 

S ample Repre sentati vene ss 115 

v 




Table of Contents — Continued 



CHAPTER 

Respondent Demographics 118 

Results of Research Questions 121 

Research Question One 121 

Research Question Two 124 

Research Question Three 137 

V. DISCUSSION 147 

Response Rate 148 

Discussion of Research Questions 149 

Research Question One 149 

Research Question Two 150 

Research Question Three 154 

Response Wave 155 

Implications 155 

Limitations 156 

Recommendations 157 

Use an Experimental Design 157 

Vary Response Wave Conditions 158 

Anecdotal Comments and Multiple Response Use 158 

Response Quality 159 

Cognitive Interviewing 159 

A Theory of Mail and Web Survey Design Issues 160 

REFERENCES 162 



vi 




Table of Contents — Continued 



APPENDICES 

A. Human Subjects Institutional Review Board Approval Letter 189 

B. Packet for School Principals 190 

C. Packet for Survey Participants 193 

D. Mail Survey 197 

E. Web Survey 213 



vii 




LIST OF TABLES 



1. Naming Convention for Survey Mode and Response Wave Response 

Condition Variables 67 

2. Respondent Group Comparison Combinations 67 

3. Definitions and Measurement Levels for All Study Variables 74 

4. Demographics of Accessible Population 89 

5. Descriptive Statistics for Response Rate by Survey Mode and Response 

Wave 115 

6. Actual and Expected Frequencies for Response Rates by Survey Mode 

and Response Wave 1 15 

7. Chi-square Goodness-of-Fit Analysis of Sample Representativeness 117 

8. Descriptive Statistics for Gender and Ethnicity by Survey Mode and 

Response Wave 119 

9. Descriptive Statistics for Age and Professional Employment by Survey 

Mode and Response Wave 120 

10. Actual and Expected Frequencies for Gender by Survey Mode and 

Response Wave 122 

1 1 . Actual and Expected Frequencies for Ethnicity by Survey Mode and 

Response Wave 123 

12. Summary of Results for Research Question One 125 

13. Descriptive Statistics for Pronoun Use by Survey Mode and Response 

Wave 126 

14. Descriptive Statistics for Pronoun Use in Anecdotal Comments by 

Response Wave 127 

15. Descriptive Statistics for Item Nonresponse by Survey Mode and 

Response Wave 128 

16. Descriptive Statistics for Response Extremity by Survey Mode and 

Response Wave 129 

viii 




List of Tables — Continued 



17. Descriptive Statistics for Yea-Saying by Survey Mode and Response 

Wave 130 

18. Descriptive Statistics for Item Completion Errors for Questions 10 and 14 

by Survey Mode and Response Wave 131 

19. Descriptive Statistics for Multiple Response Use in Mail Surveys by 

Response Wave 132 

20. Descriptive Statistics for Response Length by Survey Mode and Response 

Wave 133 

21. Descriptive Statistics for the Square Root of Response Length by Survey 

Mode and Response Wave 134 

22. Descriptive Statistics for Response Length in Anecdotal Comments by 

Response Wave 135 

23. Descriptive Statistics for Response Equivalency by Survey Mode and 

Response Wave 136 

24. Summary of Results for Research Question Two 138 

25. Descriptive Statistics for Variables Used in Logistic Regression 

Models 140 

26. Pearson Correlations Between Continuous Predictor Variables 141 

27. Analysis of Effects Prior to Entry into the Logistic Regression Model 

Predicting Mail Survey Mode, All Predictors Included 142 

28. Pinal Logistic Regression Model Predicting Mail Survey Mode from Age, 
Response Length, Gender, Yea-Saying, Ethnicity, and Response 

Extremity 144 

29. Analysis of Effects Prior to Entry into the Logistic Regression Model 

Predicting Wave 1, All Predictors Included 146 



IX 




LIST OF FIGURES 



1. Sample Item Combining Short Answer and Forced-Choice Response 

Options 77 

2. Sample Item with Multiple Sub-Items 78 

3. Sample Item-in-a-Series Format 79 

4. Front Cover of Mail Survey 80 

5. Back Cover of Mail Survey 81 

6. Screenshot of Web Survey Opening Page 82 

7. Sample Formatting Differences Between Identical Items in the Mail 

(Top) Versus the Web (Bottom) Survey 84 

8. Survey Questions 45, 46, and 48 Used to Analyze Pronoun Use 102 

9. Sample Item with Percent Scaled Response 104 

10. Sample Item with Yes and No Responses 104 

11. Survey Questions 10, 12 and 18 106 

12. Sample Contingency Item, Question 14 106 

13. Examples of Partial Items for Which Mail Respondents Provided 
Multiple Responses and the Rules Used to Determine the 

Retained Response 110 

14. Variations of Words in Open-Ended Responses that Were Assigned 

Specific Word Length V alues Ill 

15. Sample Item with Percent Scaled Response Options 1 12 



x 




CHAPTER I 



INTRODUCTION 

Chapter 1 presents an introduction to the current study and is arranged into seven 
sections. The first section presents a statement of the problem. The second section 
provides background information regarding the context of the study. The third section 
presents the research questions to be analyzed. The fourth section presents infonnation 
regarding the significance of the research. The fifth section addresses delimitations of the 
study. The sixth section presents key terms and definitions used throughout this 
document. The last section provides an overview of this document’s contents. 

Statement of the Problem 

Survey research contributes a plethora of data to advance American education 
(Ary, Jacobs, & Razavieh, 2002) *, stimulating changes in practice, policy, and funding 
and enhancing our understanding of those working in and being served by our nation’s 
educational system (Borkan, 2006). Mail surveys are a popular means to gather survey 
research data; however, as technology, computers and the Internet improve, mail surveys 
are increasingly being used in conjunction with (or even replaced by) web surveys 
(Dillman, 2000; Kiesler & Sproull, 1986; Schonlau, Fricker, & Elliot, 2002). Mail and 
web surveys have many similarities. They can be designed to look similar, contain 
identical items, and gather the same infonnation (Dillman, 2000). Both are also visual, 
self-administered, can be completed at a respondent’s leisure, and involve reading items 
and writing responses (Dillman, 2000). Thus, users of both modes may believe that they 
are compatible and that data from the two modes can be aggregated. Research shows, 

1 All references in this dissertation follow APA style as expressed in the American 
Educational Research Journal. 
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however, that there are important differences between mail and web surveys that may 
influence the populations represented and the quality of data obtained, suggesting that the 
two modes may not be equivalent (Kiesler & Sproull, 1986). Thus, studies examining 
data quality in mail and web mixed-mode designs are needed (Dillman, 2000; Dillman et 
ah, 2001; Dillman & Tamai, 1988; Schonlau et ah, 2002). 

Having multiple data collection waves is also a commonality in survey research; 
however, it is not studied often and results are conflicting (Sobal & Ferentz, 1989). While 
some studies observe few or no differences between early and late responders (Bostick, 
Pirie, Luepker, & Kofiron, 1992; Fraze, 1986; Gillispie, 1997; Goudy, 1976; Irani, Gregg, 
& Telg, 2004; Sobal & Ferentz, 1989), others have reported differences in demographics 
(Dallosso et al., 2003) and response quality (Bostick et ah, 1992). 

True experimental survey research designs are not always possible or practical in 
a real-world educational setting (Fraenkel & Wallen, 2006). Constraints such as time, 
money, resources, and survey users’ skills may limit the use of experimental designs. 
Additionally, such designs are not always appropriate when the study’s purpose is 
exploratory, providing important and timely data for local use, rather than making 
inferences and generalizations to populations (Fraenkel & Wallen, 2006). Exploratory 
survey research studies with nonexperimental designs are common (e.g., Kittleson, 1995; 
Paolo, Bonaminio, Gibson, Partridge, & Kallail, 2000; Schaefer & Dillman, 1998; 
Schleyer & Forrest, 2000; Schuldt & Totten, 1994; Tse et al., 1995; Walsh, Kiesler, 
Sproull, & Hesse, 1992; Witte, Amoroso, & Howard, 2000; Zhang, 2000), as are studies 
in which respondents self-select the survey administration mode (Dillman, West, & 

Clark, 1994; Shih & Fan, 2007b; Parker, 1992; Walsh et al., 1992). Given that data from 
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these studies are used to make important local decisions, it is crucial that researchers have 
a better understanding of how surveys behave in these circumstances (Dillman, 2000). 

Despite the surge of studies involving Internet surveys in the last decade, Internet 
survey research is still in its infancy, with “very little decisive empirical evidence from 
which to draw definitive conclusions about the optimal design and employment of 
Internet surveys” (Schonlau et ah, 2002, p. 74). They also argue, “little is known about 
the effects of web survey instrument design on how survey participants respond to a 
particular survey question. . .or what sort of design enhances. . .information accuracy” 
(Schonlau et al., 2002, p. 79). 

The current study was designed to answer calls for further research on mixed- 
mode designs and their effect on respondent demographics and data quality; fill in gaps in 
the field’s understanding of the effects of subsequent data collection waves on respondent 
demographics and response quality; and finally, expand the literature involving designs 
with survey mode self-selection. 

Background 

Few technologies have become as quickly and widely used in contemporary 
society as computers and the Internet. As Internet technology and the software to access 
the web advance and the number of users increases, social science researchers are seizing 
opportunities to use the web to conduct surveys for a variety of purposes including 
marketing, advertising, public opinion polling (Couper, 2000; McCullogh, 1998; Mosley- 
Matchett, 1998), and psychological testing (Denner, 1977; Elwood, 1969; Space, 1981). 
Electronic surveys, including both email and web surveys, have important advantages 
over other traditional survey modes such as lower costs, resources, dissemination time, 
and response time (Kiesler & Sproull, 1986; Oppennan, 1995; Parker, 1992; Schaefer & 
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Dillman, 1998; Schmidt, 1997; Schuldt & Totten, 1994; Shannon & Bradshaw, 2002; 
Smith, 1997; Weible & Wallace, 1998), as well as decreased response bias (Martin & 
Nagao, 1989; Mazzeo & Harvey, 1988). Compared to email surveys, web surveys offer 
additional benefits including broader stimuli potential (color, audio, video, animation, 
graphics, 3D), automated piping and error-checking features (versing, complex 
branching, skips, pop-up messages, and features to standardize responses), and automated 
electronic data collection, entry and analysis (Dillman, 2000; Tourangeau, 2004; Weible 
& Wallace, 1998). 

Given their benefits, web surveys also provide important challenges for 
researchers as compared to other survey modes, including sampling issues, lower 
response rates, lower response consistency, and technical issues (Borkan, 2006; Dillman, 
2000). Web survey response rates are usually lower than mail response rates (Dillman & 
Bowker, 2001; Shih & Fan, 2007b), with some exceptions (Cobanoglu, Warde, & Moreo, 
2001; Irani et al., 2004; Jun, 2005; Parker, 1992; Pettit, 2002; Smee & Brennan, 2000; 
Wu, 1997), and sometimes even lower than email response rates (Guterbock, Meekins, 
Weaver, & Fries, 2000; Kwak & Radler, 2002; Matz, 1999). Sample representativeness is 
a concern among web surveys because online and offline populations differ (Borkan, 
2006; Dillman, 2000; Schaefer & Dillman, 1998; Swoboda, Muehlberger, Weitkunat, & 
Schneeweiss, 1997; Tse, 1998). 

Given the expected increase in their use, it is important for social science 
researchers to understand the advantages, limitations, and influences of web survey 
research, yet much research is still needed as the field is relatively young. The earliest 
studies using computers and email for data collection were conducted around the mid- 
1980’s (Crawford, 1982; Erdman, Klein, & Greist, 1983; Kiesler & Sproull, 1986; Nyce 
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& Groppa, 1983; Sproull, 1986; Sproull & Kiesler, 1986; Steinfeld, 1983). The earliest 
studies involving web surveys in particular are even more recent (Beniger, 1998; Kiesler 
& Sproull, 1986; McCullogh, 1998; Mosley-Matchett, 1998; Schmidt, 1997; Smith, 

1997; Stanton, 1998). 

Researchers suggest that it may be best to study web surveys by comparing them 
to other survey modes such as mail, email, and telephone surveys (Dillman, 2000; Kiesler 
& Sproull, 1986; Schaefer & Dillman, 1998) to eliminate the weaknesses of web surveys 
(Dillman, 2000). Such “mixed-mode” studies may be implemented sequentially (e.g., 
initially delivering a mail survey and then delivering an email or web survey in a follow- 
up reminder) or multiple modes may be administered simultaneously, permitting 
respondents to self-select the survey mode (Borkan, 2006; Dillman, 2000; Dillman & 
Tarnai, 1988; Shih & Fan, 2007b). While such designs may increase response rate and 
sample representativeness, they introduce a new concern regarding whether the data can 
be aggregated, thus studies comparing mode effects on data quality are needed (Dillman, 
2000; Dillman et ah, 2001; Dillman & Tarnai, 1988; Schonlau et ah, 2002). Social 
science researchers note that questionnaire design, target population, respondent 
demographics, data quality and data types may be influenced by the mode by which 
survey data are gathered (Babbie, 2007; Dillman, 2000; Tourangeau, 2004; Tourangeau, 
Rips, & Rasinski, 2000). Many studies have been conducted to examine web surveys in 
mixed-mode designs (e.g., Borkan, 2006; Cobanoglu et al., 2001; Fiala, 2004; Guterbock 
et al., 2000; Hancock & Flowers, 2001; Underwood, Kim, & Matier, 2000), often 
producing conflicting results (Schaefer & Dillman, 1998; Shih & Fan, 2007b). 
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Theoretical Framework 



The current study sought to examine the effects of mail and web surveys on 
demographics and response quality using a mixed-mode design with mode self-selection. 
The theoretical framework for the current study is nestled in the interface between web 
and mail survey theories as they are generally understood in the social sciences, with 
specific emphasis on respondent demographics and response quality. The current study 
was also expanded to include an analysis of demographics and response quality among 
early and late responders. 

Demographics 

The literature regarding gender effects on mail and web surveys is inconsistent. 
Among electronic surveys, some studies report more males than females (Graphics 
Visualization and Usability Center [GVUC], 1999; Schmidt, 1997; Sheehan & Hoy, 
2000), while others report more females than males (Ayers, 2004; Borkan, 2006; 
McCabe, Boyd, Couper, Crawford, & D’Arcy, 2002; Witte et al., 2000). Kiesler and 
Sproull (1986) note that while there were more males than females in their randomly- 
sampled study of 15 1 university students and faculty/staff, “. . .gender. . .did not affect 
response rates” (p. 408). In mail and web surveys among college students, two studies 
observed more females than males in both mail and web modes because so many more 
females responded (McCabe et al., 2002; Underwood et al., 2000). A similar distribution 
was reported by Hayslett and Wildemuth (2004) among 300 academic reference 
librarians, but the distribution by mail and web modes was not significant, a < 0.05. 
Borkan (2006) confirmed this finding in a randomized mail and web survey of 2,000 
Ohio middle and high school teachers in which differences among mail and web 
respondents were not statistically significant. 
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Many studies suggest that more Caucasians than other ethnic groups respond to 
surveys regardless of the mode (GVUC, 1999; McCabe et ah, 2002), but especially for 
web surveys (Berry, 2006; Witte et ah, 2000). A nationwide web survey using a stratified 
random sample among 1,201 school counselors produced a response rate of 19% (n = 
231), comprised of 87% females and 88% Caucasians (2% African-American, 1% 
American Indian, 1% Asian, and 5% Hispanic) (Berry, 2006). In contrast, Smith & 
Leigh’s (1997) study about sexual fantasies among undergraduate students observed no 
differences in ethnicity. 

Regarding age, the literature suggests that web survey respondents are of the same 
age as mail respondents (Hayslett & Wildemuth, 2004; Smith & Leigh, 1997), or younger 
(Pahnquist & Stueve, 1996; Schmidt, 1997; Zhang, 2000). In their study of lesbians, gays 
and bisexuals regarding therapist selection, Kaufman et al. (1997) observed that 
electronic respondents were more disclosing of their sexual preferences, more educated, 
had higher incomes, and were younger than mail respondents. Schmidt (1997) observed 
that web respondents were younger and male with higher socioeconomic status and 
education. No age differences were revealed in Hayslett and Wildemuth’s (2004) study 
among 300 reference librarians and in Smith and Leigh’s (1997) study regarding 
undergraduate students’ sexual fantasies. 

Very few studies of professional experience effects by survey mode were found in 
the literature. In studies of American urologists, fewer years of physician practice and 
clinical experience were observed among Internet respondents as compared to mail 
respondents (Hollowell, Patel, Bales, & Gerber, 2000; Kim et al., 2000). Hayslett and 
Wildemuth’s (2004) study among 300 academic reference librarians reported that 92% of 
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respondents had a Master of Library Science degree and 71% had worked in libraries for 
10 or more years. 

Response Quality 

Some studies suggest that there are no differences in response quality between 
mail and electronic modes (Matz, 1999; Mehta & Sividas, 1995; Tse et ah, 1995; Weible 
& Wallace, 1998; Yun & Trumbo, 2000), and no differences between mail and web 
modes in particular (Dillman et ah, 2001; Hancock & Flowers, 2001; Saphore, 1999; Yun 
& Trumbo, 2000). Other studies have observed differences (Perkins & Yuan, 2001), 
suggesting that web surveys have better response quality than mail surveys (Weible & 
Wallace, 1998). 

In Kiesler and Sproull’s (1986) study of pronoun use, they counted the number of 
personal pronouns in three open-ended items. The items solicited respondents’ responses 
regarding their most recent illness, personal habits that are bothersome, and things that 
elicit personal pride and satisfaction. Their analyses revealed no differences by survey 
mode among 151 university students and faculty/staff (Kiesler & Sproull, 1986). 

Item nonresponse, one of the most commonly studied response quality 
characteristics, is a measure of the number of items on a survey questionnaire that were 
skipped but should have been answered (Pettit, 2002). In studies comparing mail and 
electronic surveys (assuming parallel surveys), some studies reported higher item 
nonresponse among mail surveys (Kiesler & Sproull, 1986) while others reported higher 
item nonresponse in electronic surveys (Howes & Mailloux, 2001; Mehta & Sividas, 
1995; Paolo et al., 2000; Webster & Compeau, 1996). Two studies observed no 
differences among mail and email surveys (Tse, 1998; Tse et al., 1995). Comparing web 
and mail surveys, web surveys have been shown to have the same (Pealer, 1999) or lower 
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item nonresponse than mail surveys (Kerwin, Brick, Levin, O’Brien, & Cantor, 2006; 
Kiesler & Sproull, 1986; MacElroy, Mikuski, & McDowell, 2002; Schaefer & Dillman, 
1998; Stanton, 1998; Truell, Bartlett, & Alexander, 2002); however, higher item 
nonresponse in web surveys has also been reported (Ahlstrom, 2004; Smee & Brennan, 
2000; Jun 2005). For example, Jun (2005) observed 2 1 .44 times more skipped items in 
web than in mail surveys. Several studies observed no differences in mail and web item 
nonresponse (Cobanoglu et ah, 2001; Matz, 1999; Pettit, 2002). Schaefer and Dillman 
(1998) provide a review of studies that found conflicting results. Unlike mail surveys, 
web surveys can be programmed so that branching, error checking, and complex skip 
patterns occur automatically and pop-up windows can provide immediate feedback 
allowing respondents to fix incorrect or skipped responses before their survey is 
submitted (Schaefer & Dillman, 1998; Tourangeau, 2004). Features such as these are 
believed to increase a respondent’s attention to survey items, producing lower item 
nonresponse in web surveys compared to other modes. On the other hand, they may 
annoy and overburden respondents, increasing item nonresponse and survey abandonment 
(Dillman, 2000; Dillman, Tortora, & Bowker, 1998; Smee & Brennan, 2000). 

Item completion error is a measure of the number of errors on a survey 
questionnaire and happens when a respondent provides an invalid response (e.g., a 
response where one is not expected, a response that cannot be coded, illegible 
handwriting) (Pettit, 2002). Sometimes this is the result of a branching or data entry 
instruction that was not understood or followed correctly (Dillman, 2000). The literature 
is inconsistent, with some studies reporting no difference in item completion errors 
among mail and web surveys (Pealer, 1999; Smee & Brennan, 2000), and others reporting 
fewer item completion errors in web compared to mail surveys (Weible & Wallace, 1998; 
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Pettit, 2002). Smee and Brennan (2000) found no differences in their comparison of mail, 
email, and web modes. Kiesler and Sproull’s (1986) health-related survey among 151 
university students and faculty/staff reported fewer item completion errors in the email as 
compared to the mail survey. Of 53 items containing response errors, 5.3% were made by 
mail respondents and 0.0% was made by email respondents (Kiesler & Sproull, 1986). 

Response length is a measure of the number of words in a respondent’s response 
to an open-ended item. It is assumed that a longer response is indicative of better 
response quality because respondents provide more data (Schaefer & Dillman, 1998). 
Several studies have observed that web surveys surpass paper/mail surveys in producing 
higher quality, more sophisticated, and longer responses to open-ended items (Kiesler & 
Sproull, 1986; MacElroy et ah, 2002; Nicholls, Baker, & Martin, 1997; Schaefer & 
Dillman, 1998; Wu, 1997). In a web and mail survey involving mode self-selection (n = 
1,228), a mean of 48.2 words was revealed for the web mode compared to a mean of 32.0 
words for the mail mode for a “Use this space to add some final thoughts” open-ended 
item (MacElroy et ah, 2002). 

Yea-saying is the tendency of a respondent to agree with item statements or 
questions independent of the item’s content (Greenleaf, 1992) as evidenced by a response 
that is located near or at the affirmative anchor of a list of response options (Kiesler & 
Sproull, 1986). Two studies of yea-saying were found, both revealing no differences by 
survey mode (Kiesler & Sproull, 1986; Weijters, Schillewaert, & Geuns, 2004). In a study 
conducted by Kiesler and Sproull (1986) comparing paper and electronic surveys, yea- 
saying was measured among 151 randomly-selected university students and faculty/staff 
using five forced-choice items. The items solicited participants’ attitudes on health- 
related topics using a seven-point Likert response set where “Agree” equals one and 
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“Disagree” equals seven. ANOVAs revealed no differences by survey mode (Kiesler & 
Sproull, 1986). 

Response extremity occurs when a respondent favors (or avoids) the extreme 
anchors (e.g., “strongly agree” or “strongly disagree” in a Likert scale) of a response scale 
(Greenleaf, 1992; Kiesler & Sproull, 1986; Naemi, 2006; Pettit, 2002). There is 
considerable variation in the way response extremity is measured in the literature (Naemi, 
2006), with some studies counting use of both extreme anchors in Likert-type response 
sets (Berg & Collier, 1953; Booth-Kewley, Edwards, & Rosenfeld, 1992; Greenleaf, 

1992; Pettit, 2002; Soueif, 1958; Sproull, 1986), and others counting use of only the most 
extreme positive response option (Brengelmann, 1960). Most studies reported no 
difference in response extremity among mail and electronic survey modes (Booth-Kewley 
et ah, 1992; Dillman et al., 2001; Helgeson & Ursic, 1989; Petit, 2002), with one 
exception (Sproull, 1986). Pettit (2002) observed no differences between web and paper 
modes in a comparison of personality questionnaires. Booth-Kewley et al.’s (1992) study 
among 246 Navy recruits also reported no differences between a computer survey with 
backtracking (e.g., respondents could return to previous pages and change their 
responses), a computer survey with no backtracking, and a paper survey. In contrast, 
SproulTs (1986) survey among 60 business professionals in a Fortune 500 company 
reported more extreme responding in email than in paper surveys. This study examined 
48 hypothetical task items with response anchors scaled from zero to 10 where zero and 
10 were coded as extreme responses (Sproull, 1986). 

Response equivalency is a measure of differences in the mean scores among items 
or collapsed item groups having ordinal, interval, or ratio level data. Researchers suggest 
that responses across surveys modes in general are not equivalent (Dillman, 2000; 
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Sudman, Bradburn, & Schwarz, 1996) due to differential effects on responses resulting 
from each mode’s degree of impersonality, perception of legitimacy, and level of 
cognitive burden imposed on the respondent (Tourangeau et ah, 2000, p. 20). A review of 
the literature revealed one study comparing paper and email that observed a difference in 
response equivalency (Miller, Daly, Wood, Brooks, & Roper, 1996), but most mail and 
electronic survey studies reported no differences (Bachmann, Elfrink, & Vazzana, 1996; 
Booth-Kewley, et ah, 1992; Hayslett and Wildemuth, 2004; Helgeson & Ursic, 1989; 
Mehta & Sividas, 1995; Pettit, 2002). In a study comparing personality questionnaires in 
web and paper modes, Pettit (2002) observed no differences in mean scale scores and no 
differences in internal consistency by survey mode. Helgesen and Ursic (1989) examined 
the equivalency of decision processes among 126 randomly assigned undergraduate 
business students via electronic and paper surveys, observing no statistically significant 
differences in response equivalency and item ordering effects by survey mode. Booth- 
Kewley et al.’s (1992) study among 246 Navy recruits reported no difference in response 
equivalency between paper, computer with backtracking, and computer without 
backtracking surveys. 

The current study also sought to examine multiple response use as well as 
response length and pronoun use among anecdotal comments. Pettit (2002) defined 
multiple response use as follows: “The total number of items for which. . .two or more 
options were selected” (p. 52). Hayslett and Wildemuth (2004) noted in their study 
among 300 academic references librarians that it occurred in their mail and web survey 
due to the use of check boxes in the mail survey versus radio buttons in the web survey. 

In the web survey, the use of radio buttons prevented respondents from selecting more 
than one response option; however, some mail survey respondents checked two or more 
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responses on their mail surveys. Two studies were found in the literature that mentioned 
multiple response use, but one discarded the multiple responses (Hayslett & Wildemuth, 
2004) while the other (Pettit, 2002) analyzed it in as one of five types of errors measured 
collectively as response errors. The current study sought to examine this phenomenon in 
greater detail by creating a variable to count the number of responses that would have 
otherwise been discarded. 

Hayslett and Wildemuth (2004) noted another problem in their study, the use of 
anecdotal comments by mail respondents but not by web respondents. They explained 
that some mail respondents wrote notes on their paper surveys regarding items they 
thought were ambiguous or didn’t understand, to describe how they interpreted an item, 
and to clarify their responses. The web survey design, however, prevented web 
respondents from writing such comments. They suggested adding text boxes at key points 
throughout the web survey to capture anecdotal comments, but noted that web 
respondents would still not be able to draw lines, circle portions of items, or write 
nontextual annotations as mail respondents could (Hayslett & Wildemuth, 2004). The 
current study created variables to analyze anecdotal comments for response length and 
pronoun use. 

Early and Late Responders 

“Early responders” refers to survey participants who respond to an initial request 
to participate in a survey, while “late responders” refers to those who participate only 
after one or more reminders (Bostick et ah, 1992). Studies have reported few or no 
differences between early and late responders in demographics (Bostick et ah, 1992; 
Fraze, 1986; Gillispie, 1997; Goudy, 1976; Irani et ah, 2004; Sobal & Ferentz, 1989) and 
substantive questions (Bostick et al., 1992; Goudy, 1976; Sobal & Ferentz, 1989), while 
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others have reported differences in demographics (Dallosso et ah, 2003) and response 
quality (Bostick et ah, 1992). Combining demographics and response quality, Sobal and 
Ferentz (1989) conducted a national mail survey among 1,010 physicians to examine 
differences in demographics, substantive questions, and response quality among the first 
and second mailings. Results after an 82% return rate revealed no differences by response 
wave in demographics (age, gender, type of residency, or region of the country), 
substantive questions regarding respondents’ residency, use of “don’t know” responses, 
or item nonresponse. They concluded that responses to the second mailing did not change 
the sample’s representativeness or the study’s substantive results (Sobal & Ferentz, 

1989). 

Research Questions 

The objective of the current study was to compare the demographic characteristics 
and response quality among mail and web surveys when respondents are allowed to self- 
select the survey administration mode. Specifically, this study addressed the following 
research questions: 

1 . Does a difference exist in demographic characteristics by survey mode or 
response wave? 

a) Does a difference exist in gender by survey mode or response wave? 

b) Does a difference exist in ethnicity by survey mode or response wave? 

c) Does a difference exist in age by survey mode or response wave? 

d) Does a difference exist in professional employment by survey mode or 
response wave? 

2. Does a difference exist in response quality by survey mode or response wave? 
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a) Does a difference exist in pronoun use by survey mode or response 
wave? 

b) Does a difference exist in pronoun use in anecdotal comments by 
response wave? 

c) Does a difference exist in item nonresponse by survey mode or 
response wave? 

d) Does a difference exist in response extremity by survey mode or 
response wave? 

e) Does a difference exist in yea-saying by survey mode or response 
wave? 

f) Does a difference exist in item completion errors by survey mode or 
response wave? 

g) Does a difference exist in response length by survey mode or response 
wave? 

h) Does a difference exist in the response length of anecdotal comments 
by response wave? 

i) Does a difference exist in response equivalency by survey mode or 
response wave? 

j) Does a difference exist in multiple response use by response wave? 

3. Do demographic characteristics and response quality predict survey mode or 

response wave? 

a) Do demographic characteristics and response quality predict survey 
mode? 
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b) Do demographic characteristics and response quality predict response 



wave? 

Significance of the Research 

Dillman notes, “The newness of Internet (e-mail and Web)... surveys means that 
virtually no research has addressed the potential mode differences that might be 
associated with these methods.” As self-administered methods, “. . .a priority in research 
is to determine the extent to which these methods. . .may mirror paper self-administered 
surveys” (2000, p. 232). Several studies comparing mail and web surveys have been 
conducted since Dillman’ s call for further research (Jun, 2005; Kaplowitz, Hadlock, & 
Levine, 2004; Kwak & Radler, 2002; Ladner, 2003; McCabe, et. ah, 2002; McElroy, et. 
al., 2002; Mertler, 2003; Sedwick, 2003; Shannon & Bradshaw, 2002; Yun & Trumbo, 
2000); however, these studies have produced conflicting results. Even fewer studies have 
examined the effects of allowing respondents to self-select the survey mode (McElroy, et. 
al., 2002; Yun & Trumbo, 2000). Yun & Trumbo (2000) conducted a survey involving 
respondent mode self-selection that compared the cost, sampling representativeness, 
response rate, and mean values of variables across mail, email, and web surveys. They 
observed that multi-mode surveys improved the representativeness of a sample without 
biasing results (Yun & Trumbo, 2000). A more recent study involving mode self-section 
compared web and mail surveys with respect to item response rates for closed-ended 
items and the length and quality of responses for open-ended items (McElroy, et. al., 
2002). This study observed that web responses have a greater number of words, 
characters, unique concepts, and sentences per response than mail survey responses. A 
Flesch-Kincaid Grade Level analysis of those survey responses shows that web responses 
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were written at the 9th grade level and mail responses were written at the 8th grade level 
(McElroy, et. ah, 2002). 

Among the important values of contemporary American society is choice. Giving 
people access to options for them to participate is paramount in a democratic society. This 
is evident in the choices offered to American voters casting ballots for public elections: 
punch cards, paper-and pencil in-person, paper-and-pencil absentee, mark sense (bubbles 
filled in and read with an optical scanner), electronic using a keyboard or a touch-screen 
monitor (data are not collected via the Internet), and the Internet (pilot-tested by the 
military during the 1996 presidential election) (Napoli, 2006). But, does this translate to 
survey research? Given the expected rise in the use of web surveys alone as well as in 
combination with other modes, it’s important to understand their effects. The results of 
the current study will be useful to survey researchers and practitioners who make 
important decisions based on data collected with mail and web surveys. Hence this study 
is significant for the following reasons: 

1 . Given that survey technologies are rapidly advancing, this study contributes 
important and timely empirical data necessary to advance the field’s 
understanding of web surveys as they compare to mail surveys. 

2. This study contributes empirical data to the understanding of mixed-mode 
survey methodologies, especially to the understanding of the effects on data 
quality in studies with mode self-selection. This is important because the 
literature is inconclusive in this area. 

3. This study responds to calls for further research regarding which survey modes 
are preferred in specific populations when participants are given the choice 
(Shih & Fan, 2007b), as well as calls for further research comparing survey 
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modes (Dillman, 2000). When multi-mode designs are not possible, these 
findings may help researchers and practitioners make infonned decisions 
regarding which mode is best. 

4. This study is unique in that no other studies were found to have been 
conducted with this population using this design. While similar designs have 
been used in other social science fields such as business and market research, 
this kind of study will fill a gap in the field of survey research conducted 
among public school guidance counselors. Knowledge in this area is important 
given the size and diversity of the population, the expected increase in the use 
of web surveys in studies involving them, and the impact that the population 
has on American education. 

5. Findings from this study are useful to researchers and practitioners conducting 
nonexperimental survey research using combined mail and web administration 
modes. Understanding the effects of respondent self-selection is important 
because randomly assigning respondents to survey modes is not always 
practical or possible in social science research. 

Delimitations 

Findings from this study should be interpreted within the context of the following 
delimitations: 

1 . The current study used a dataset derived from a larger study that had already 
been completed. As such, the current study’s design was restricted by the 
circumstances of the larger study. Nonrandom sampling and survey mode self- 
selection limited generalizability and the instrument’s item content and 
structure prevented an analysis of instrument reliability. Survey items were 
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fixed, possibility producing an order effect among responses. Additionally, the 
current study’s dataset was stripped of all identifying variables to ensure 
respondents’ confidentiality. Thus, it was not possible to contact respondents 
to clarify vague or incomplete responses, verify data, or conduct cognitive 
interviews. 

2. School principals verified the names of participants on the sampling frame list 
and disseminated surveys to participants. While it is assumed that principals 
provided accurate data and disseminated surveys in a consistent and timely 
manner, this cannot be confirmed. 

3. As their initial contact, participants received a mail survey with the web 
survey’s URL address version printed on the cover. This was necessary 
because participants’ emails were not available. Web survey respondents had 
to take the extra initiative to locate a computer, connect to the Internet, and 
type in the URL address to access the web survey. 

4. It’s assumed that web survey respondents completed their surveys using 
computers with the appropriate hardware, software, and Internet connectivity 
for the survey to display correctly; however, this cannot be confirmed. 

5. It’s assumed that web respondents were sufficiently computer literate to locate 
the survey web site, navigate within a web page, provide responses using a 
keyboard, and submit the survey electronically; however, this cannot be 
confirmed. 

6. While this study used a common instrument in two different modes to 
examine mode effects, the instrument itself may affect results. 
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7. This study is valid only for the population, setting, and time period defined 



herein. 



Definitions 

The following terms have been defined to promote a better understanding of their 

use in the context of the current study. 

Anonymous: A respondent’s identity cannot be determined by anyone, including the 
researcher (Babbie, 1990). 

Branching: Instructions on a questionnaire that provide direction to an interviewer or 
respondent regarding how to move through a questionnaire (Alreck & Settle, 
1995). 

Browser: Computer software that receives and interprets hypertext data facilitating 

navigation of the World Wide Web and Internet and the displaying f web pages. 
Examples of browser software include Microsoft Internet Explorer and Netscape 
Navigator. 

Census: Studying all members of a population rather than sampling a portion of the 
population to represent the whole (Alreck & Settle, 1995). 

Click: Placing a computer mouse pointer over an action button on a computer screen, and 
then pushing the mouse button to initiate an action. 

Client: A software program that facilitates access to a server from another computer. 

Computer-Assisted Personal Interview (CAPI): Face-to-face interview in which the 

interviewer asks questions that appear on a portable computer, and then types the 
respondent’s answers into the computer during the interview. 

Computer-Assisted Self-Administered Interview (CASI): Interview in which a respondent 
reads questions on a computer screen and types responses into the computer in the 
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absence of an interviewer. Also “Computerized Self- Administered 
Questionnaire.” 

Computer-Assisted Telephone Interview (CATI): Interview in which an interviewer calls 
a respondent on the telephone, asks the respondent questions that appear on a 
computer screen, and types the respondent’s answers into the computer during the 
interview (Babbie, 2007). 

Computer-Based Survey: A survey that uses any type of computer (mainframe, desktop, 
laptop, hand-held, etc.) to gather survey responses and may or may not be 
connected to the Internet. Responses are automatically entered into a database. 

Confidential: Survey in which a respondent’s identity can be detennined by at least one 
member of a research team, but the respondent is guaranteed that his/her identity 
will be kept a secret (Babbie, 1990). 

Contingency Item: A survey item presented to all respondents, that is then followed by a 
sub-item for only those respondents who provided a specific response (Babbie, 
2007). 

Demographics: Conditions or attributes that are assigned to people to facilitate putting 
people into groups (Alreck & Settle, 1995). In survey research, demographics are 
used to determine if specific groups are represented by a sample and to make 
comparisons among groups. Examples include race/ethnicity, age, gender, marital 
status, income, education, employment, and socioeconomic status. 

Email (Electronic Mail): Messages that can be exchanged electronically from one 

computer to another (usually over the Internet or an intranet) via some kind of 
connection such as a dial-up telephone line, a cable, or a wireless connection. 
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Email Survey: Survey that involves a questionnaire typed into the text of an email 

message that is disseminated to and returned from a respondent electronically via 
a computer and the Internet or an intranet. Respondents participate by editing the 
original email message to include their responses. The researcher manually enters 
responses into a database for analysis. 

Face-to-Face Interview: Interviews in which an interviewer and respondent are in each 
other’s presence while the interview is being conducted and during which the 
interviewer records the respondent’s responses. 

Forced-Choice Items: Survey items that present a statement or question to which the 
respondent provides an answer by selecting from among a list of responses 
supplied by the researcher (Babbie, 2007). True/false and multiple choice and 
examples of forced-choice items. Also called “closed-ended” items. 

Hardware (Computer Hardware): The physical devices and components of a computer 
(Alreck & Settle, 1995). 

Hit: A single request from a computer’s World Wide Web browser for a single item from 
a Web server. Hits are commonly counted and used to determine how often a web 
page is visited. 

Home Page: A specific web page designated as the entry point, main page or first page of 
a website. 

Host: A computer on a network that stores services that can be accessed by other 

computers on the network. Such a computer may act as a “host” for a client’s web 
survey. 
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Hyperlink: A reference point in a web page or document such as text or an image that an 
individual can click on to retrieve a document, be taken to another place within a 
page, or be taken to another website. 

Hypertext: Text that contains hyperlinks that, when clicked on, automatically forward the 
user to another place on a page, to another page within a website, or to another 
website. 

HyperText Markup Language (HTML): The coding language used to create web pages 
and hypertext documents that can be used on the World Wide Web. 

HyperText Transfer Protocol (HTTP): The protocol for moving hypertext files across the 
Internet. 

Incentive: A reward such as money, a gift certificate, a prize, a drawing entry, etc. 

designed to motivate an individual to behave in a specific way. Incentives are 
sometimes used to encourage participants to respond to a survey. 

Interactive Voice Response (IVR) Survey: Survey in which a computer calls a telephone 
number and then presents recorded questions to a respondent, to which a 
respondent responds by speaking or selecting appropriate numbers on the 
telephone’s numeric touch-tone keypad. 

Internet: The cooperative, global network of millions of linked computers that exchange 
information electronically using TCP/IP protocols. 

Internet Literacy: The essential knowledge that an individual needs in order to function 
independently on the Internet. 

Internet Protocol (IP) Address: A unique numerical address assigned to a computer that 
distinguishes one computer from others when communicating via the Internet. 

Internet Service Provider (ISP): An entity that provides Internet access for individuals. 
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Interview: A face-to-face or telephone data collection encounter in which an interviewer 
asks questions of a respondent (Babbie, 2007). 

Intranet: A network within an organization that uses the same mechanism as the Internet, 
but is for organization members only and is closed to public access. 

Item Completion Error: Instances in which a respondent provides an incorrect response or 
a response where one was not expected. 

Item Nonresponse: Instances in which a respondent skips an item that should have been 
answered. 

Likert Scale: A type of scale (usually five-point) used to rate a respondent’s level of 
agreement or disagreement with a statement (Alreck & Settle, 1995). 

Link: A phrase containing numbers and characters (e.g., http://www.wmich.edu) that is 
the World Wide Web address to a web page. 

Listserve: An electronic mailing list. 

Mail Survey: A paper questionnaire that is delivered and returned via a mail carrier such 
as the United States Postal Service or interoffice mail. Also referred to as 
“traditional,” “postal,” “hard-copy,” “surface mail,” “paper,” “paper-and-pencil,” 
and “snail mail” surveys. 

Multi-Mode Survey: A survey that involves any two or more of a variety of available 

questionnaire fonnats, such as interview, computer-assisted, email, web, and mail 
formats. 

Newsgroup: An Internet resource where people post and read messages related to a 
specific topic. 

Nonresponse Rate: Survey nonresponse rate is a measure of the number of respondents 
who did not participate in a survey divided by the total number of participants 
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invited to participate (Fraenkel & Wallen, 2006). Item nonresponse rate is a 
measure of the number of items on a survey questionnaire that a respondent 
skipped but should have answered (Fraenkel & Wallen, 2006). 

Open-Ended Items: Survey items that present a statement or question to which the 
respondent writes an answer in their own words (Babbie, 2007). 

Probe or Probing: An interview technique in which an interview uses nondirective 

phrases or questions (e.g., Anything more?) to solicit a more complete response to 
a respondent’s incomplete or vague response (Babbie, 2007). 

Questionnaire: A document containing items used to solicit data from individuals for 
analysis (Babbie, 2007). 

Reliability: The repeatability of a measure. 

Respondent: An individual who provides data to be included in a research study by 
responding to items on a survey questionnaire (Babbie, 2007). 

Response Equivalency: A comparison of the mean scores of selected items or collapsed 
item groups to detennine if statistically significant differences exist in the means. 

Response Extremity: A respondent’s use of the most extreme response option among a 
range of forced-choice responses in a Likert or Likert-type scale. 

Response Length: The number of words written in response to an open-ended item. 

Response Quality: The extent to which survey responses contribute accurate and 
informative data to a study. 

Response Rate: The percentage of useable surveys that are returned (or people 

participating in an interview) divided by the number of surveys delivered (Babbie, 
2007). 
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Self- Administered Questionnaire: A questionnaire that is read and completed by a 

respondent in the absence of an interviewer and without the assistance of a survey 
administrator. 

Server: A computer or software application that provides a service to other computers. In 
survey research, the server is where the questionnaire is located and is “served” to 
a respondent when the respondent accesses it. 

Socially Desirable Response: Response bias that results when an individual’s response is 
based on what the individual believes is socially acceptable or provides a better 
impression of themselves rather than on what is really true. Also called social 
desirability bias. 

Software (Computer Software): The programs, coded instructions, and applications of a 
computer (Alreck & Settle, 1995). 

Survey: A research design in which a sample of individuals selected from a population is 
systematically questioned and then results are analyzed and generalized back to 
the population (Alreck & Settle, 1995). 

Survey Mode: The manner by which a survey is administered to respondents (e.g., mail, 
email, web, face-to-face interview, telephone interview, etc.). 

Telephone Survey: Survey in which an interviewer calls a respondent on the telephone, 
asks the respondent each survey question, and records the respondent’s answers. 
Also called a telephone interview. 

Transmission Control Protocol/Internet Protocol (TCP/IP): Transmission Control 

Protocol converts messages from their source into electronic data packets for 
transmission and then back into messages at their destination. Internet Protocol 
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handles addressing functions that permit electronic data packets delivered across 
multiple networks to be sent and received correctly. 

Uniform Resource Locator (URL): The unique address of an Internet resource (e.g., a web 
page) that is located on the World Wide Web. 

Web Page: A document on the World Wide Web with a unique URL address. 

Web Survey: An electronic survey questionnaire created on the Internet or World Wide 
Web as a web page. Respondents access the survey’s web page, respond to items 
as they appear on the computer screen, and then submit the survey electronically. 
Web surveys may include color, audio, video, animation, graphics, complex 
branching, and skip patterns, pop-up messages, pull-down menus, checkboxes, 
radio buttons, fdl-in text boxes, scrolling windows, and features to standardize 
responses. Also referred to as “web-based,” “online,” or “Internet” surveys. 

World Wide Web (WWW): A hypertext-based Internet server system that maintains 
html-fonnatted data including text, graphics, audio, video, and databases. 

Yea Saying: The number of instances in which a respondent selects a positive or “yes” 
response. 

Organization of the Dissertation 

Chapter 1 of this dissertation provides an overview of the study. It introduces the 
problem, provides the background, lists the research questions, explains the significance 
of the research, addresses delimitations, and provides a list of definitions for key terms 
used throughout this document. Chapter 2 presents a review of the literature. It presents a 
brief history of survey research, describes some of the main characteristics of surveys, 
compares survey administration modes, and then reviews the literature relative to the 
study’s research questions. Chapter 3 describes the methodology used in the study. It 
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describes the study’s design, explains the use of a dataset drawn from a larger study, 
reviews the research questions, introduces the variables with their constitutive and 
operational definitions, describes the survey instrumentation, details the data collection 
and sampling procedures, and finally explains the analyses used to answer the research 
questions. Chapter 4 presents the study’s findings for each research question. Finally, 
Chapter 5 presents a discussion of the study’s findings. It reviews the results from 
Chapter 4 in the context of the literature, describes the study’s limitations, and then 
addresses practical implications and recommendations for future research. 
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CHAPTER II 



LITERATURE REVIEW 

Chapter 2 presents an introduction to the current study and is arranged into three 
sections. The first section presents a brief history of survey research. The second section 
describes some of the characteristics of survey research, such as questionnaire design, 
costs, incentives, response rate, and a comparison of early and late responders. The third 
section describes survey research modes. This section compares self-administered and 
interview surveys, discusses modes effects and the use of mixed-mode mail and web 
surveys, and concludes with respondent demographics and response quality. 

Brief History of Survey Research 

Surveys typically administer a questionnaire to a sample selected from a 
population, and then use that data to describe, explain or predict the attitudes, knowledge, 
skills, opinions, feelings, behavior, needs, affiliations, demographics, or lifestyles of the 
population (Alreck & Settle, 1995; Babbie, 2007; Fink, 1995; Fink, 2006; Fraenkel & 
Wallen, 2006; Schonlau et ah, 2002). The practice of using surveys and censuses has 
been noted throughout history dating back as far as ancient Egyptian civilizations where 
rulers used censuses to gather data about their subjects (Babbie, 1990; Babbie, 2007). 

One of the first political attitudinal surveys was employed in 1880 by Karl Marx, a 
Gennan political sociologist, who mailed 25,000 questionnaires to French workers to 
solicit their views regarding employer exploitation (Babbie, 1990; Babbie, 2007). 

The individuals credited as the pioneers of contemporary survey research are 
Samuel A. Stouffer and Paul F. Lazarsfeld (Babbie, 1990). Stauffer’s work focused on 
applying empirical research methods to study social phenomena, contributing much 
contemporary survey methodology’s design, sampling, instrumentation, and analysis. 
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Lazarsfeld also advanced the application of empirical methods to social research, but his 
main contributions include using mechanized data processing equipment to analyze 
survey data, and the development of permanent centers such as Columbia University’s 
Bureau of Applied Social Research to study and advance the field of survey research 
(Babbie, 1990). 

Babbie (1990) notes that contemporary survey research is mainly a product of 20 th 
century American research in three sectors of American society: the U.S. Bureau of the 
Census, commercial and political polling companies, and scientific survey research by 
American universities. The U.S. Bureau of the Census has made important contributions 
to survey sampling, design, and data collection. Best known for the U.S. Census, which 
has been conducted every 10 years since 1790, the Census Bureau uses additional surveys 
to collect data between censuses, and maintains current demographic and economic data 
that are used by researchers in a variety of marketing and social service fields (Babbie, 
1990). Today, surveys are used for policymaking, planning, assessment, and evaluation 
(Fink, 2006), and comprise the most common method of data collection in education 
research (Ary et al., 2002). 

Surveys have also advanced with technology. Telephone surveys were popular in 
the 1960s and 1970s as telephones became more common in households (Tourangeau, 
2004). Computer-based surveys such as computer-assisted telephone interviewing (CATI) 
and computer-assisted personal interviewing (CAPI) followed in the 1970s through 1990s 
as the use of desktop and laptop computers increased (Couper et al., 1998; Tourangeau, 
2004). The earliest studies using computers and email for data collection were conducted 
around the mid-1980’s (Crawford, 1982; Erdman et al., 1983; Kiesler & Sproull, 1986; 
Nyce & Groppa, 1983; Sproull, 1986; Sproull & Kiesler, 1986; Steinfeld, 1983). Today, 
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interactive voice response (IVR) surveys are common and, with the advent of the Internet 
and Word Wide Web, email and web surveys are common and continually advancing. 
Survey research has become a multibillion dollar industry, with much attention devoted to 
understanding its methodologies and applications. 

Several national and international professional associations exist today that use 
annual conferences and peer-reviewed journals to study emerging survey research trends 
and methodologies, including the American Association for Public Opinion Research, 
American Statistical Association, Centre for Survey Research and Methodology (ZUMA) 
in Germany, International Association of Survey Statisticians, Statistical Society of 
Canada, and World Association of Public Opinion Research. Associations that provide 
conferences and journals for examining survey research relevant to specific fields include 
the American Educational Research Association, American Evaluation Association, 
American Marketing Association, American Political Science Association, American 
Sociological Association, Association for Research on Nonprofit Organizations and 
Voluntary Action, and Visitor Studies Association among others. 

Survey Research Characteristics 

In discussions of survey research, the term “survey” is used interchangeably to 
describe a survey instrument such as a questionnaire, as well as to describe the entire 
process of surveying from planning to conduct a survey to reporting results. The process 
of survey research is comprised of several steps: defining survey research objectives, 
identifying the type of data to be collected and the precision of results, specifying a 
population, developing a sampling frame, detennining a sample size, selecting a sample 
using random or nonrandom methods, selecting one or mode survey response modes (e.g., 
mail, web, interview), designing items, designing and field testing a survey instrument, 
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disseminating the survey dissemination and follow-up reminders, and finally collecting, 
coding, cleaning, reducing and analyzing data (Babbie, 1990; Babbie, 2007; Dillman, 
2000; Fink, 2006; Fraenkel & Wallen, 2006; Schonlau et ak, 2002). Additional 
considerations in survey research include response rate, data quality, questionnaire 
features (e.g., appearance, length, item design), timeliness of results, cost, resource 
requirements (e.g., staff, equipment, supplies, postage, printing, training, travel), 
confidentiality, ethics, and politics (Babbie, 1990; Wu, 1997). Compared to other 
research methods, survey research tends to be less expensive, permits greater 
standardization in data collection, and permits sampling a large population (Babbie, 

2007). 

Questionnaire Design 

The design of a questionnaire and the order of items are critical to obtaining a 
good response rate with accurate data (Babbie, 2007; Bishop & Smith, 2001; Dillman, 
2000). Objective items in a survey questionnaire should be placed at the beginning, while 
sensitive and demographic items should be placed near the end (Fink, 2006). In a study of 
the impact of item order, Benton and Daly (1991) found that less educated respondents 
were more influenced by item order than more educated respondents. Babbie (2007) 
suggests that items should be designed according to the following guidelines: items (e.g., 
force-choice vs. open-ended items) should be appropriate for the study, be clear and 
precise, ask about only one topic at a time, be relevant to the respondent, be as short as 
possible, be stated in non-biased and non-negative terms, and respondents must be both 
competent and willing to answer the items (p. 245-251). The questionnaire should also 
contain clear instructions and should be field-tested and revised before use (Babbie, 2007; 
Fraenkel & Wallen, 2006). Overall questionnaire length needs to be appropriate for 
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respondents, aesthetically appealing, and easy to complete (Fink, 2006). A questionnaire 
length of no more than four pages is suggested for optimum response rates (Heberlein & 
Baumgartner, 1978; Yammarino, Skinner, & Childers, 1991). 

The survey in the current study was designed with general objective items at the 
beginning, more content-specific items in the middle, and demographic items at the end. 
Instructions and items were designed following suggested guidelines and the survey was 
field-tested prior to use. The length of the mail survey, excluding the front and back cover 
pages which contained no survey items, was 14 pages, each 8.50 x 7.00 inches. 

Costs 

All surveys, regardless of mode, have financial costs that researchers must 
consider. Depending on the survey mode, costs can be incurred for human labor, postage, 
paper and printing, telephone calls, and computer equipment, among others. Costs can 
also be influenced by sample size, study complexity, time, the availability of resources, 
and researcher skills. Survey researchers must carefully weigh the costs of each survey 
mode against potential benefits (e.g., response time, data quality) given study populations, 
circumstances and constraints in order to determine the most appropriate survey mode. 

Human labor is a substantial, but often over-looked expense. Holding all other 
survey factors constant, human labor costs associated with survey design, participant list 
compilation, and data analysis vary little across survey modes, but do differ across mode 
for instrument dissemination and data entry (Schonlau et ah, 2002). Regarding personnel 
costs, the most expensive surveys to conduct are face-to-face surveys followed by 
telephone surveys due to interviewer labor. Both modes incur expenses associated with 
training interviewers and conducting interviews. Face-to-face surveys (and sometimes 
telephone surveys) also have costs associated with scheduling interviews. Face-to-face 
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surveys have additional costs associated with interviewers’ travel time. Mail surveys 
don’t have interviewer costs, but do have labor costs associated with assembling survey 
packets as well as printing and mailing questionnaires. Face-to-face interviews, telephone 
interviews, mail and email surveys each require data entry clerks to code responses and 
then enter data into an electronic format for analysis. 

Holding survey sample size constant, other costs across survey mode are lesser 
concerns and vary with survey complexity (Schonlau et ah, 2002). Costs for interviewer 
mileage expenses for face-to-face surveys, phone calling expenses (including long 
distance and CATI programming) for telephone surveys, and printing, paper supplies, and 
postage expenses for mail surveys seem to be similar overall (Schonlau et al., 2002). 
Schonlau et al. (2002) note that the labor cost for one day of a researcher’s time for a mail 
survey corresponds to the cost of printing and mailing several hundred questionnaires 
costing three to four dollars each (p. 1 1-12). If preparing a survey budget, Weible and 
Wallace (1998) suggest planning for the following costs per 100 participants with 
$20. 00/hour for human labor and $0. 05/printed page: $215 for a mail survey, $113 for a 
faxed survey, and $77 for either an e-mail survey or web survey. 

Costs in Mail and Electronic Surveys . Electronic (email and web) surveys are generally 
less expensive than mail surveys (Weible & Wallace, 1998), because they use less paper, 
postage and human labor. Email and web surveys minimize or eliminate costs associated 
with paper, labels, envelopes, printing, postage, and human labor to prepare surveys for 
mailing and process returned surveys. Several studies show that mail surveys are more 
expensive than email surveys (Bachmann et al., 1996; Kiesler & Sproull, 1986; Mertler, 
2002; Parker, 1992; Schaefer & Dillman, 1998; Sproull, 1986), but determining the costs 
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for human labor in email surveys may be difficult to calculate and easy to overlook 
(Bachmann et al, 1996). 

Web survey use has exploded due to its low costs relative to mail, face-to-face 
interview and computer-assisted telephone surveys, as well as its ability to quickly 
process, return, and analyze massive amounts of data (Dillman & Bowker, 2001; Yun & 
Trumbo, 2000). Web surveys have even been shown to be less expensive than email 
surveys (Mertler, 2002; Watt, 1999). Web surveys that compile data electronically 
substantially reduce or eliminate costs associated with converting data into an electronic 
format for analysis, reducing or eliminating human data entry error. Watt (1999) found 
that the typical cost per 10,000 respondents is $0.65 for a web survey versus $1.64 for an 
email survey, both less expensive than a mail survey, and the cost of electronic surveys 
per respondent decreases as the sample size increases. He notes, however, that initial 
human labor costs for a web survey may be higher than mail surveys due to programming, 
questionnaire design, and computer network maintenance (Watt, 1999). Mertler (2002) 
observed a total cost of $ 120 for a web survey and $4,000 for a mail survey. In one multi- 
mode study comparing mail, email and web surveys, electronic survey costs were higher 
compared to mail as the electronic survey involved a full mailing as well as electronic 
delivery, and considerable time was spent developing and administering the web survey 
and managing email (Yun & Trumbo, 2000). Weible and Wallace (1998) argue that the 
convenience and lower cost benefits of electronic surveys can make up for their decreased 
response rates. 

Incentives 

The literature on incentives suggests that response rate increases when 
respondents are given incentives for their participation (Babbie, 2007; Dillman, 
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Carpenter, Christensen, & Brooks, 1974; Dillman, 2000). Examples of incentives include 
money, gifts or gift certificates, donation to a charity, lottery prize drawings, and copies 
of research reports. Some researchers suggest that providing incentives may affect 
response quality or sample representativeness, while others argue there is no effect. More 
research is needed to detennine the effect of incentives. Due to the large sample size 
(A=2,880) in the current study, cash and gift incentives were not provided for survey 
respondents. Rather, respondents were infonned of the importance of their comments to 
the study and were given an opportunity to receive a copy of the final report. 

Response Rate 

The response rate is a measure of the number of individuals returning completed 
surveys divided by the number of individuals in a sampling frame, expressed as a 
percentage (Babbie, 2007). Response rate is an important value reported in survey 
research studies because it’s an indicator of a sample’s representativeness and the validity 
of its findings (Rylander, Propst, & McMurty, 1995). A higher response rate is generally 
associated with lower nonresponse error (Babbie, 1990; Underwood et ah, 2000), thus 
increasing generalizability and confidence that the sample accurately reflects the 
population (Rylander et ah, 1995). Nonresponse error is the error created by individuals 
in a sample who do not respond to a survey, but if they had responded, would have 
provided different answers than those of individuals who did respond (Fraenkel & 

Wallen, 2006; Groves, 1989). 

Researchers desire to achieve the highest response rates possible within given 
constraints (e.g., time, cost, resources), believing that low response rates threaten a 
survey’s utility. In a study of theoretical models of survey participation, Groves and 
Couper (1998) have shown that respondents and nonrespondents differ systematically. 
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However, some studies challenge this belief (Curtin, Presser, & Singer, 2000; Keeter, 
Miller, Kohut, Groves, & Presser, 2000). Keeter et al. (2000) compared the results of two 
identical telephone surveys conducted under different levels of effort. The “Standard” 
survey was conducted among a nonrandom sample over a five-day period, and the 
“Rigorous” method was conducted among a random sample over an eight-week period 
and included a $2.00 incentive. Response rates were 36% for the standard method and 
61% for the rigorous procedure. Results for 91 comparisons revealed that the two surveys 
produced similar results with an average difference of 2% among responses, mainly in 
demographic items, and no differences exceeding 9%. Curtin et al.’s (2000) study 
comparing the responses of surveys with 60-70% response rates to those with 20-40% 
response rates revealed minimal differences in substantive responses. 

Response Rates for Mail and Electronic Surveys . Studies reviewing the literature report 
that email response rates vary widely (Sheehan & McMillan, 1999) from as low as 6-19% 
to as high as 73% (Oppennan, 1995; Weible & Wallace, 1998), and mail survey response 
rates vary from 27 to 56 percent (Weible & Wallace, 1998). In a comparison of mail and 
electronic (email and web) response rates by mode, electronic survey response rates are 
generally believed to be similar to (Smith, 1997; Truell et al., 2002; Vazzana & 
Bachmann, 1994) or lower than (Bachmann et al., 1996; Bachmann, Elfrink & Vazzana, 
1999; Dillman & Bowker, 2001; Dillman et al., 2001; Fiala, 2004; Hayslett & 

Wildemuth, 2004; Hollowell et al., 2000; Jones & Pitt, 1999; Kiesler & Sproull, 1986; 
Kim et al., 2000; Kittleson, 1995; Matz, 1999; Mertler, 2003; Paolo et al., 2000; Schuldt 
& Totten, 1994; Shih & Fan, 2007a; Shih & Fan, 2007b; Tse, 1998; Tse et al., 1995; 
Underwood et al., 2000; Weible & Wallace, 1998) mail surveys. Web survey response 
rates in particular are usually lower than mail response rates (Dillman & Bowker, 2001; 
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Quigley, Riemer, Cruzen, & Rosen, 2000; Shih & Fan, 2007b), although Jun’s (2005) 
survey about health behaviors among 1,000 undergraduate university students revealed no 
difference in mail (38.3%) and web (35.2%) survey response rates. 

To compare the response rate effect of paper versus email invitations to web 
surveys, Hayslett and Wildemuth (2004) conducted a survey among 300 randomly 
sampled North Carolina academic reference librarians. One hundred participants were 
randomly assigned to each of three groups: mail survey with a mail invitation, web survey 
with a mail invitation, and web survey with an email invitation. Results revealed response 
rates of 43.2% (n = 51) usable surveys for the mail mode with mail invitation, 22.9% (n = 
28) for the web survey with paper invitation, and 33.1% (n = 39) for the web survey with 
email invitation. Differences between the three formats were statistically significant, p < 
0.05 (Hayslett & Wildemuth, 2004). 

A recent meta-analysis of 35 studies comparing mail and email surveys reported 
that email responses rates were 20% lower than mail response rates (Shih & Fan, 2007a). 
A meta-analysis of 43 studies comparing mail and web surveys reported that web 
response rates were 8% lower using an unweighted mean (14% lower using a weighted 
mean) than mail response rates (Shih & Fan, 2007b). Concerning just mail surveys, a 
meta-analysis of 31 studies from 1986 to 2000 suggests there has been a steady decline in 
mail survey response rates from an average of 46% in 1995-96 to an average of 3 1% in 
1998-99 (Sheehan, 2001). It appears, though, that despite a possible decrease in mail 
survey response rates, mail surveys still exceed electronic survey response rates. 

Web survey response rates that exceed mail response rates are less common in the 
literature (Cobanoglu et al., 2001; Irani et al., 2004; Parker, 1992; Pettit, 2002; Smee & 
Brennan, 2000; Wu, 1997), possibly due to studies using populations more familiar with 
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technology and the Internet (Kaminer, 1997; Wu, 1997), as well as using populations 
with a high number of undeliverable email addresses (Schuldt & Totten, 1994). For 
example, a survey of 140 AT&T employees worldwide produced response rates of 60% 
for email and 38% for mail (Parker, 1992). This same study noted a mode preference 
effect when respondents self-selected the mode: 28% of respondents invited via e-mail 
subsequently returned their responses by mail (Parker, 1992). 

To increase survey response rates, some researchers suggest using mixed-mode 
surveys, such as combining web and mail modes, to provide respondents with more 
response options (Dillman, 2000; Dillman et ah, 2001; Groves et ah, 2004; Schaefer & 
Dillman, 1998; Shettle & Mooney, 1999). Unfortunately, there is no evidence to support 
that using mail and web surveys simultaneously (e.g., pennitting mode self-selection) 
produces any overall increase in response rates (Dillman et ah, 1994; Quigley et ah, 

2000). Rather, it results in minimal use of the web mode (Schonlau et ah, 2002) or even 
reduced overall response rate (Griffin, Fischer, & Morgan, 2001). Dillman et al. (1994) 
observed no response rate improvements when respondents to the 1990 census short fonn 
were offered either a mail-only mode (71% response rate) or a mail and interview mode 
with mode self-selection (69% response rate). Switching modes in follow-up contacts, 
however, has been shown to improve response rates (Dillman, Dillman, & Makela, 1984; 
Paxson, Dillman, & Tarnai 1995; Schonlau et al., 2002; Shettle & Mooney, 1999). Shettle 
and Mooney (1999) suggest that it is better to use a sequential rather than concurrent 
mixed-mode strategy, such that participants are initially contacted in one mode, with 
follow-ups to nonrespondents made in other modes. Their national survey of college 
graduates with a 68% response rate after mail contacts, increased to 81% after telephone 
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interviews were offered, and then increased to 88% after in-person interviews were 
offered. 

A recent meta-analysis of 43 mail and web mixed-mode studies with mode self- 
selection revealed that response rates were higher in mail than in web modes, and 
respondents were inclined to select the mode by which they were initially contacted (Shih 
& Fan, 2007b). Among the 43 studies, the unweighted average mail response rate of 27% 
exceeded the web response rate of 19% by 8%. When respondents received a mail survey 
as their first contact with the option to complete a web survey, response rate for the mail 
survey was 16% greater than for the web survey. Alternatively, when respondents 
received a web survey as their first contact with the option to complete a mail survey, the 
response rate was 29% greater for the web survey compared to the mail survey (Shih & 
Fan, 2007b). 

The current study used a mail and web mixed-mode design with a mail survey as 
the initial contact. The population of school counselors was not known to have any 
exceptional familiarity with technology and the Internet. Thus, it was expected that there 
would be more mail than web respondents. 

Survey Nonresponse . The survey nonresponse rate is a measure of the number of 
respondents who did not participate in a survey divided by the total number of 
participants invited to participate (Fraenkel & Wallen, 2006). Survey nonresponse can 
result when participants refuse to participate (e.g., lack of time, interest, or knowledge; 
unwillingness to be surveyed), are unable to participate (e.g., scheduling conflicts, illness, 
disability, illiteracy), or cannot be reached (e.g., incorrect address or telephone number, 
not home) (Fraenkel & Wallen, 2007). 
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Factors that may reduce survey nonresponse include establishing (or already 
having) a relationship between respondents and a sponsoring organization, a respondent’s 
personal interest in the survey topic, and a plain instead of a fancy web survey layout 
(Dillman et ah, 1998), as well as timeliness of the topic (Dillman, 2000). In a meta- 
analysis of mail surveys, Yammarino et al. (1991) found that respondents also expect 
postage to be provided and its absence may deter a response. Other ways to reduce survey 
nonresponse include assuring confidentiality and anonymity, using appropriately designed 
questionnaires and items, providing postage for mail surveys, starting with easy and 
nonthreatening questions, using call-backs, mailing/emailing reminders and replacement 
surveys, and offering incentives (Fraenkel & Wallen, 2006). Surveys with government 
sponsorships have been shown to produce higher response rates than surveys without 
such sponsorships (Dillman, 2000; Dillman, Singer, Clark, & Treat, 1996; Heberlein & 
Baumgartner, 1978), possibly due to an “appealing to authority” effect (Groves, Cialdini, 
& Couper, 1992). 

Shifts in society’s willingness to participate in surveys may be related to growing 
concerns about privacy (Goree & Marzalek, 1995), confidentiality, telemarketing, and 
trust (Babbie, 2007; Singer, 2003; Singer, Mathiowetz, & Couper, 1993), as well as 
respondent’s interest in the subject matter (Kawasaki & Raven, 1995). Singer (2003) 
notes that 13% of respondents who were willing to participate in her survey were 
unwilling to sign the consent fonn. Individuals are especially concerned about 
computerized data collection methodologies (Glasner, 1999; Martin & Nagao, 1989; 
Rosenfeld & Booth-Kewley, 1996). Americans are investing in ways to protect their 
identities, screen callers (Link & Oldendick, 1999), and restrict visitors (Blakely & 
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Snyder, 1997). Over eight million American’s live in gated communities and almost 40% 
of new residential developments are gated (Blakely & Snyder, 1997). 

One of the most important factors in decreasing survey nonresponse may be the 
number and type of contacts. A meta-analysis of web surveys suggests that follow-up 
contacts with nonrespondents, personalized contacts, and pre-notification are dominant 
factors influencing response rates (Cook, Heath, & Thompson, 2000). For email surveys, 
one, two and three contacts with respondents produced 29%, 41% and 57% response rates 
respectively (Schaefer & Dillman, 1998). Fowler (1993) warns that surveys that don’t 
include follow-up procedures have lower response rates. Dillman (2000) suggests the 
following to improve response rates: 1) provide a respondent-friendly questionnaire; 2) 
provide four first-class mail contacts plus a “special” contact; 3) provide return envelopes 
with first class stamps (i.e., not business reply envelopes); 4) personalize correspondence; 
and 5) provide financial incentives (pp. 150-153). Researchers should also obtain as much 
demographic information as possible on sampling frame participants to pennit an 
examination of sample representativeness. 

In the current study, attention was devoted to providing a respondent-friendly 
questionnaire that included funder and sponsor logos, researcher contact infonnation, and 
the respondent’s name and school; the initial mailing was followed with four first-class 
reminders; and postage-paid return envelopes were provided (business reply envelopes 
were used because it was not practical to use “real” stamps). 

Early and Late Responders 

“Early responders” refers to survey participants who respond to an initial request 
to participate in a survey, while “late responders” refers to those who participate only 
after one or more reminders (Bostick et ah, 1992). Studies have reported few or no 
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differences between early and late responders in demographics (Bostick et al., 1992; 
Fraze, 1986; Gillispie, 1997; Goudy, 1976; Irani et al, 2004; Sobal & Ferentz, 1989) and 
substantive questions (Bostick et al., 1992; Goudy, 1976; Sobal & Ferentz, 1989), while 
others have reported differences in demographics (Dallosso et al., 2003) and response 
quality (Bostick et al., 1992). 

Regarding demographics, Gillispie’s (1997) study among 172 (59% response rate) 
randomly selected business faculty at four North Carolina universities examined the use 
and perceived value of computer-mediated communication for instructional purposes. 
Results revealed no differences among early and late responders by gender, job title, 
highest degree, age and teaching experience (Gillispie, 1997). Fraze (1986) also 
compared demographics by response wave but found one significant difference. This 
study among 653 Future Farmers of America (FFA) participants explored the relationship 
between FFA participation in high school, and career choice and job satisfaction after 
graduating. Following cluster random sampling, the survey response rate was 44% (n = 
290), 165 early responders and 125 late responders. Analyses using /-tests revealed a 
statistically significant difference between early and late responders for high school grade 
point average, but not for educational level, FFA offices held, judging contest 
participation, leadership contest participation, career choice, or job satisfaction. Early 
responders had higher high school grade point averages than late responders, p < 0.05 
(pp. 34-35). 

Another study of high school graduates found that early responders have higher 
socioeconomic status than late responders (Pavalko & Lutterman, 1973). This study 
conducted a mail survey among 10,321 parents of Wisconsin high school graduates 
soliciting data on their child’s educational and occupational activities following 
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graduation. The response rate was 91% after the initial mailing and three reminders. Four 
response waves were compared, one corresponding to each mailing. Results revealed that 
the parents of children with higher socioeconomic status (as measured by their child’s 
high school rank, intelligence, educational plans, educational and occupational 
attainment) were 20% more likely to respond during the first response wave (Pavalko & 
Lutterman, 1973). Another study examining demographics was conducted among 35,131 
randomly selected participants (63.5% response rate). In this nationwide mail survey 
about urinary health, results revealed that late responders were less likely to be women 
(OR 0.88, 95% Cl 0.81-0.96) and younger (OR 0.99, 95% Cl 0.98-0.99), but more likely 
to be South Asian than Caucasian (OR 1.92, 95% Cl 1.66-2.22) (Dallosso et ah, 2003). 

Combining demographics and response quality, Sobal and Ferentz (1989) 
conducted a national mail survey among 1,010 physicians to examine differences in 
demographics, substantive questions, and response quality among the first and second 
mailings. Results after an 82% return rate revealed no differences by response wave in 
demographics (age, gender, type of residency, or region of the country), substantive 
questions regarding respondents’ residency, use of “don’t know” responses, or item 
nonresponse. They concluded that responses to the second mailing did not change the 
sample’s representativeness or the study’s substantive results (Sobal & Ferentz, 1989). 
Bostick et al. (1992) conducted a 20-minute telephone survey about risk factors for 
cardiovascular disease among 24 1 randomly sampled physicians from the upper Midwest 
states, 172 early responders and 69 late responders. Findings revealed no significant 
differences in demographics (e.g., gender, year of graduation from medical school, field 
of practice, percent of time devoted to primary care, average number of patients seen 
daily, and city size) or study variables, but a significant difference was revealed in social 
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desirability responding,/; = 0.03. Early responders provided more socially desirable 
responses than late responders (Bostick et al., 1992). 

The current study sought to examine the effect of response wave on demographics 
and response quality at two data collection cycles. Wave 1 comprised the initial mailing 
and first three reminders and Wave 2 comprised only the fourth reminder. The decision 
was made to split the response waves in this manner to examine the effect of extending a 
survey’s data collection into a new school year. Additionally, splitting in this manner 
pennitted a comparison among two data cycles lasting the same number of months. The 
first response wave occurred during the first three months of data collection (April - July 

2005) and the second response wave occurred during the last three months (September - 
December 2005). No reminders were sent in the time between the two response waves. 

Survey Research Modes 
Self- Administered Versus Interview Surveys 

Researchers generally categorize surveys into two types based on their data 
collection methods: interviews and self-administered questionnaires (Babbie, 2007; Fink, 

2006) . Interviews are conducted face-to-face or via telephone or teleconference wherein 
the respondent communicates personally with an interviewer. Self-administered surveys 
include mail, email and web surveys in which respondents read item questions or 
statements on their own and provide their own written responses. Interview-based surveys 
have additional costs and considerations over other self-administered modes such as 
training and paying interviewers, interviewer safety, and telephone and transportation 
expenses (Babbie, 1990; Fink, 2006). Unlike self-administered surveys where 
respondents provide their own responses, interviews use interviewers to gather and record 
data from respondents. Thus, it is essential that interviewers are neutral, that they use 



45 




consistent questioning and probing (eliciting clarification for incomplete or vague 
responses) methods, that they record responses accurately, and that their presence has no 
effect on a respondent’s responses (Babbie, 2007). Telephone surveys can be less 
expensive than face-to-face interviews and can be automated pennitting more control 
over data collection (Babbie, 2007; Fraenkel & Wallen, 2006). Computer-aided telephone 
interviewing (CATI), computer-aided personal interviewing (CAPI), and random digit 
dialing are some of these technological advances. 

The advantages of interviews over self-administered surveys include fewer 
incomplete questionnaires, fewer misunderstood items and responses, higher response 
rates, and greater sampling flexibility (Babbie, 2007). Response rates in telephone 
surveys in general, however, are declining, possibly due to growth in telemarketing 
(Babbie, 2007). The advantages of telephone interviews over face-to-face interviews 
include less cost and time, greater safety for the interviewer, and the interviewer has 
fewer effects on respondent’s responses (Babbie, 2007). 

Self-administered surveys such as direct administration to a group, mail, web, and 
email (no studies were found that administered surveys via cell phones) are those that 
respondents complete by hand or by computer on their own. Such surveys are typically 
disseminated to respondents via the mail, facsimile, personal delivery, publications, group 
meetings, email or the Internet. In surveys directly administered to groups (such as during 
a meeting), a researcher has access to all participants at the same time and in the same 
place. In a mail survey, a paper survey is mailed directly to a respondent and then mailed 
back to the researcher by the respondent. Among the advantages of mail surveys are that 
they can be used to reach participants that are difficult to reach by telephone or face-to- 
face, respondents can complete them at their leisure, and respondents can to take 
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sufficient time to provide thoughtful responses (Fraenkel & Wallen, 2006). Among their 
disadvantages, they require respondents who are literate, there is no opportunity for 
building rapport (as in interviewing), and there is no (or limited) opportunity for a 
researcher to clarify vague or incomplete responses. Both email and web surveys are 
disseminated and returned electronically via the Internet. 

Self-administered surveys (excluding surveys directly administered to a group) 
typically have lower response rates than interviews and require a series of follow-up 
reminder contacts and replacement questionnaires (Babbie, 2007). Thus, Babbie (2007) 
suggests that users monitor daily and cumulative return rates throughout the data 
collection cycle. Self-administered surveys that are directly administered to a group, 
however, have among the highest response rates (Fraenkel & Wallen, 2006). Advantages 
of self-administered surveys over interviews are cost, speed, lack of interviewer bias, and 
increased privacy (Babbie, 2007). Although web surveys are less expensive to conduct 
than mail surveys, they are less representative of the population than mail surveys 
(Babbie, 2007). 

Survey Mode Effects 

Social science researchers note that questionnaire design, target population, 
respondent demographics, data quality and data types may be influenced by the mode by 
which survey data are gathered (Babbie, 2007; Dillman, 2000; Tourangeau, 2004; 
Tourangeau et al., 2000). Respondents provide different responses to identical items in 
different survey modes due to a respondent’s perception of a mode’s degree of privacy, 
legitimacy, and cognitive burden (Tourangeau, 2004; Tourangeau et al., 2000). For 
example, telephone interviews are perceived to be less private than computer assisted 
modes, long questions or response option lists may overburden a respondent’s memory, 
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the interview may be rushed (Tourangeau, 2004; Tourangeau et ah, 2000), or respondents 
may confuse legitimate telephone surveys with telemarketing (van Leeuwen & de Leeuw, 
1999). Dillman (2000) describes a framework for administering effective mail and 
Internet surveys that has been used successfully for over 25 years. This framework, called 
the “Tailored Design Method” describes a survey as a social interaction between a 
researcher and a respondent and stresses the importance of effectively communicating the 
survey to the respondent (Dillman, 2000). 

Electronic surveys, including both email and web surveys, have important 
advantages over other traditional survey modes such as lower costs, resources, 
dissemination time, and response time (Kiesler & Sproull, 1986; Opperman, 1995; 

Parker, 1992; Schaefer & Dillman, 1998; Schmidt, 1997; Schuldt & Totten, 1994; 
Shannon & Bradshaw, 2002; Smith, 1997; Weible & Wallace, 1998), and possibly 
decreased response bias (Martin & Nagao, 1989; Mazzeo & Harvey, 1988). Compared to 
email surveys, web surveys offer additional benefits including broader stimuli potential 
(color, audio, video, animation, graphics, 3D), automated piping and error-checking 
features (versing, complex branching, skips, pop-up messages, and features to standardize 
responses), and automated electronic data collection, entry and analysis (Dillman, 2000; 
Tourangeau, 2004; Weible & Wallace, 1998). 

Given their benefits, web surveys also provide important challenges for 
researchers as compared to other survey modes, including sampling issues, lower 
response rates, lower response consistency, and technical issues. Specific concerns 
involve whether web respondents are representative of the target population, whether web 
and mail responders have similar demographics, whether web and mail surveys produce 
responses of similar quality, and whether survey data gathered via multiple modes can be 
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aggregated (Dillman, 2000; Schonlau et al., 2002). Web survey use, especially 
commercial use, has continued to explode despite coverage and nonresponse issues 
(Couper, 2000), as society strives to find cheaper, faster ways to collect data (Tourangeau, 
2004). 

Mixed-Mode Mail and Web Surveys 

Researchers suggest that it may be best to study web surveys by comparing them 
to other survey modes such as mail, email, and telephone surveys (Dillman, 2000; Kiesler 
& Sproull, 1986; Schaefer & Dillman, 1998). Mixed-mode designs are expected to reduce 
the limitations of individual survey modes, reduce overall costs, and improve overall 
response rates (Dillman, 2000; Schaefer & Dillman, 1998). Schnolau, Fricker, and Elliot 
(2002) also suggest using mixed mode surveys to increase coverage when conducting 
web surveys among general populations for which email addresses are not available. 
Groves and Kahn (1979) observed evidence of respondent mode preferences in their 
national telephone interview in which 39% of respondents said they favored a telephone 
interview, 23% preferred a face-to-face interview, and 28% preferred mail. 

The use of mixed-mode surveys, however, raises the issue of whether respondents 
using different modes provide the same answers and represent the same populations 
(Dillman et al., 2001). This study will contribute important and timely infonnation 
concerning differences in the demographic characteristics of respondents and response 
quality among web and mail survey modes. 

Respondent Demographics 

Obtaining a representative sample for an electronic survey is an especially 
important concern (Dillman, 2000; Schaefer & Dillman, 1998; Swoboda et al., 1997; Tse, 
1998) because individuals in the general population have unequal access to email and the 
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Internet (Dillman, 2000). Tse (1998) notes that while almost everyone has a postal 
address, email surveys are restricted to those with email access. Email respondents tend to 
over represent the middle and upper classes (GVUC, 1999; Mehta & Sividas, 1995). 
Sheehan and Hoy’s sample of email survey respondents from an Internet population 
revealed 60% had at least a bachelor’s degree, and 86% checked their email at least once 
per day (2000). Some studies suggest that most web survey respondents have some 
college experience, are Caucasian, live in suburban areas, access the Internet daily from 
home, have an average income of $57,300, and are 38 years old (GVUC, 1999). Pitkow 
and Kehoe (1996) note that most web survey respondents are in education and computer- 
related fields, have a college or advanced degree, and have more wealth than the majority 
of the population. Researchers caution against generalizing findings to the population at 
large that are based on Internet newsgroups and other restricted sampling frames 
(Bachmann et ah, 1996; Swoboda et. ah, 1997). There are also potential sources of 
sampling bias within the Internet population associated with differences in Internet 
connectivity, equipment, browser software, and experience with the Internet (GVUC, 
1999). 

The current study will compare the gender, ethnicity, age, and professional 
employment demographics of mail and web survey respondents. All respondents in the 
current study are assumed to have Internet access; however, it’s expected that there may 
be some differences in demographic characteristics by survey mode or response wave. 
Gender 

Gender distributions for surveys in general suggest that females may respond 
more often than males, regardless of the survey mode (Borkan, 2006; Dillman et ah, 

2001; Fiala, 2004; Fritz, 2004; Green & Stager, 1986; Jun, 2005; Smith, 1983; 
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Underwood et al., 2000). This trend is also reflected in surveys involving school 
counselors in particular (Berry, 2006; Fitch & Marshall, 2004; Fritz, 2004; Ruebensaal, 
2006; Young, 2004). In a statewide mail survey of 102 systematically sampled full-time 
Illinois secondary school counselors, Fritz (2004) reported an unadjusted response rate of 
29.25%, comprised of 76% females and 24% males. 

There is less consistency in the gender distributions among mail and electronic 
surveys. Among electronic surveys, some studies report more males than females 
(GVUC, 1999; Schmidt, 1997; Sheehan & Hoy, 2000), while others report more females 
than males (Ayers, 2004; Borkan, 2006; McCabe et al., 2002; Witte et al., 2000). A study 
by Ayers (2004) among elementary music teachers examined gender effects on mail and 
web surveys with mode self-selection. Participants, invited via mail, were directed to 
either access a web survey online or request to have a paper survey mailed to them (only 
seven requests for paper surveys were received). Ayers (2004) reported a response rate of 
18.60% (n = 412), comprised of 86.90% (n = 358) females and 13.10% (n = 54) males. 

Kiesler and Sproull (1986) note that while there were more males than females in 
their randomly-sampled study of 15 1 university students and faculty/staff, 

“. . .gender. . .did not affect response rates” (p. 408). In mail and web surveys among 
college students, two studies observed more females than males in both mail and web 
modes because so many more females responded (McCabe et al., 2002; Underwood et al., 
2000). A similar finding was reported by Borkan (2006) in a randomized mail and web 
survey of 2,000 Ohio middle and high school teachers. Borkan (2006) noted that 73.86% 
(n = 80,023) of the participants in the target population were female and approximately 
60% of the respondents in the overall sample as well as in each condition were female; 
however, the gender effects by survey mode were not statistically significant. In a study of 
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151 randomly selected student and faculty email users, Kiesler and Sproull (1986) 
observed that females wrote longer responses to open-ended items than males, but found 
no gender by survey mode interaction. Another study by Hayslett and Wildemuth (2004) 
among 300 academic references librarians observed a 72% response rate for females, but 
differences by mail and web modes were not significant, p < 0.05. 

Ethnicity 

Many studies suggest that more Caucasians than other ethnic groups respond to 
surveys regardless of the mode (GVUC, 1999; McCabe et ah, 2002), but especially for 
web surveys (Witte et ah, 2000). In a comparison of web and paper/mail surveys 
collecting alcohol and drug use data among a sample of 3,606 randomly selected and 
randomly assigned university students, McCabe et al. (2002) observed more Caucasians 
(68%) than all other ethnic groups combined in the overall sample,/? < 0.001, as well as 
for the mail,/? <0.01, and web surveys,/? < 0.001, separately. A nationwide web survey 
using a stratified random sample among 1,201 school counselors produced a response rate 
of 19% (n = 23 1), comprised of 87% females and 88% Caucasians (2% African- 
American, 1% American Indian, 1% Asian, and 5% Hispanic) (Berry, 2006). In contrast, 
Smith & Leigh’s (1997) study about sexual fantasies among undergraduate students 
observed no differences in ethnicity. 

Age 

Age distributions for surveys seem to suggest that web survey respondents may be 
of the same age as mail respondents (Hayslett & Wildemuth, 2004), or younger than mail 
respondents (Kaufman et al., 1997; Pahnquist & Stueve, 1996; Schmidt, 1997; Zhang, 
2000). In their study of lesbians, gays and bisexuals regarding therapist selection, 
Kaufman et al. (1997) observed that electronic respondents were more disclosing of their 



52 




sexual preferences, more educated, had higher incomes, and were younger than mail 
respondents. Schmidt (1997) observed that web respondents were younger and male with 
higher socioeconomic status and education. Hayslett and Wildemuth’s (2004) study 
among 300 reference librarians observed no age differences between mail and web modes 
and 50% of respondents were 45-54 years of age. No differences were observed in 
undergraduate students’ age, marital status, ethnicity or education between web (n = 72) 
and paper (n = 56) modes in Smith and Leigh’s (1997) survey about sexual fantasies. 
Professional Experience 

Very few studies of professional experience effects by survey mode were found in 
the literature. In studies of American urologists, fewer years of physician practice and 
clinical experience were observed among Internet respondents as compared to mail 
respondents (Kim et ah, 2000; Hollowed et ah, 2000). Hayslett and Wildemuth’s study 
among 300 academic reference librarians reported that 92% of respondents had a Master 
of Library Science degree and 71% had worked in libraries for 10 or more years. 

Response Quality 

Response quality refers to the extent to which survey responses contribute 
accurate and infonnative data to a study (Dillman, 2000). The quality of responses is 
important in every survey mode because important decisions are made based on the data 
surveys provide. Interviews are believed to have comparatively lower item nonresponse 
and higher social desirability responding that self-administered modes (Sudman et ah, 
1996). Given the greater anonymity of electronic surveys, electronic survey respondents 
give less socially desirable responses (Kiesler & Sproull, 1986), possibly because people 
tend to be less inhibited and more self-absorbed when communicating with a computer 
(Kiesler, Siegel, & McGuire, 1984). Given the similarities between web and mail surveys 
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(e.g., visual stimulus, under respondent’s complete control, items are read and then 
responses are written), Dillman (2000) notes that web responses are more similar to mail 
responses than to interviews. 

With the rise in the use of web surveys in addition to or in place of mail surveys, it 
is important to understand response quality both within and between these modes. 
Although mail and web surveys share some features such as self-administration and 
providing written responses (e.g., handwritten or via a keyboard), there may be important 
differences in the quality of the data that each produces. Kiesler and Sproull (1986) note 
considerable similarity in response quality between paper and electronic modes, but not 
enough to consider them interchangeable. Some studies suggest that there are no 
differences in response quality between mail and electronic modes (Matz, 1999; Mehta & 
Sividas, 1995; Tse et ah, 1995; Weible & Wallace, 1998; Yun & Trumbo, 2000), and no 
differences between mail and web modes in particular (Dillman et ah, 2001; Hancock & 
Flowers, 2001; Saphore, 1999; Yun & Trumbo, 2000). Other studies have observed 
differences (Perkins & Yuan, 2001), suggesting that web surveys have better response 
quality than mail surveys (Weible & Wallace, 1998). 

Findings regarding mail and web response quality are inconclusive and difficult to 
interpret. One of the reasons for this is that there are differences in the way response 
quality is defined and measured. Under the construct “response quality,” some studies are 
actually only measuring the length of qualitative responses and item nonresponse, some 
just study response errors, and others focus on social desirability responding. Wu’s (1997) 
measures of data quality included question completion rate, comments completion rate, 
and “average comments wording rate.” Kiesler and Sproull’s (1986) study of response 
quality included item nonresponse, not following instructions, social desirability, yea- 
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saying (selecting positive responses), item response length, and item self-disclosure. In 
the current study, response quality measures include pronoun use, item nonresponse, the 
use of extreme responses, yea-saying, item completion errors, the use of multiple 
responses, response length, and response equivalency. 

Pronoun Use 

Pronoun use is measured by the number of pronouns such as “I” and “We” that 
occur in a response to an open-ended item. This measure is typically found in the 
literature in studies involving social desirability effects (Kiesler & Sproull, 1986). 
Researching social desirability effects is beyond the scope of the current study; however, 
differences between the number of pronouns among web and mail responses to open- 
ended questions was examined. In Kiesler and SproulTs (1986) study of pronoun use, 
they counted the number of personal pronouns in three open-ended items. The items 
solicited respondents’ responses regarding their most recent illness, personal habits that 
are bothersome, and things that elicit personal pride and satisfaction. Their analyses 
revealed no differences by survey mode among 151 university students and faculty/staff 
(Kiesler & Sproull, 1986). In the current study, pronoun use was measured by counting 
the number of personal pronouns in open-ended responses and then analyzed by survey 
mode and response wave using ANOVAs. 

Item Nonresponse 

Item nonresponse, one of the most commonly studied response quality 
characteristics (Pettit, 2002), is a measure of the number of items on a survey 
questionnaire that were skipped but should have been answered. Pettit (2002) provides 
the following operational definition: “the total number of items for which no response 
was provided” (p. 52). Item nonresponse can occur when a respondent doesn’t understand 
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an item or an instruction for completing an item; an item is embarrassing, irrelevant, 
vague, too complex, or too personal; the respondent doesn’t correctly follow branching 
instructions; the most appropriate response is not among those provided in forced-choice 
items (e.g., categories are not exhaustive); or an interviewer fails to record a response 
(Fraenkel & Wallen, 2006). Additionally, at the data entry stage, a data entry clerk may 
fail to record a respondent’s response or a response may not get recorded because it is 
invalid or illegible (Fraenkel & Wallen, 2006). Suggestions to reduce item nonresponse 
include designing appropriate items; reducing questionnaire or interview length; 
eliminating items that are leading, biased, threatening, or complex; providing exhaustive 
categories for forced-choice items; and pre-testing the questionnaire (Fraenkel & Wallen, 
2006). Low item nonresponse is an indicator of good survey response quality (Couper, 
Blair, & Triplett, 1999; Schaefer & Dillman, 1998; Stanton, 1998). 

In studies comparing mail and electronic surveys, some studies reported higher 
item nonresponse among mail surveys (Kiesler & Sproull, 1986) while others reported 
higher item nonresponse in electronic surveys (Howes & Mailloux, 2001; Mehta & 
Sividas, 1995; Paolo et al., 2000; Webster & Compeau, 1996). Two studies observed no 
differences among mail and email surveys (Tse, 1998; Tse et al., 1995). Locke & Gilbert 
(1995) observed no item nonresponse differences in a study of psychological assessment, 
self-disclosure, and experiential differences using computer, paper, and interview modes. 
Kiesler and Sproull’s (1986) study among 151 university students and faculty/staff noted 
that 22% of mail respondents compared to 10% of electronic survey respondents had one 
or more skipped items. Comparing web and mail surveys, web surveys have been shown 
to have the same (Pealer, 1999) or lower item nonresponse than mail surveys (Kiesler & 
Sproull, 1986; MacElroy et al., 2002; Schaefer & Dillman, 1998; Stanton, 1998; Truell et 



56 




al., 2002); however, higher item nonresponse in web surveys has also been reported 
(Ahlstrom, 2004; Smee & Brennan, 2000; Jun 2005). For example, Jun (2005) observed 
2 1 .44 times more skipped items in web than in mail surveys. Several studies observed no 
differences in mail and web item nonresponse (Cobanoglu et al., 2001; Matz, 1999; Pettit, 
2002). Schaefer and Dillman (1998) provide a review of studies that found conflicting 
results. 

Unlike mail surveys, web surveys can be programmed so that branching, error 
checking, and complex skip patterns occur automatically and pop-up windows can 
provide immediate feedback allowing respondents to fix incorrect or skipped responses 
before their survey is submitted (Schaefer & Dillman, 1998; Tourangeau, 2004). Features 
such as these are believed to increase a respondent’s attention to survey items, producing 
lower item nonresponse in web surveys compared to other modes. On the other hand, they 
may annoy and overburden respondents, increasing item nonresponse and survey 
abandonment (Dillman, 2000; Dillman et al., 1998; Smee & Brennan, 2000). 

To provide some clarification of the effects of automated features on item 
nonresponse in web surveys, a study conducted by Smee and Brennan (2000) examined 
mode effects using a 50-item questionnaire in five fonnats: mail, email, a single 
continuous web survey with no error checking, a multiple-page web survey with 
automated branching and no error checking, and a multiple-page web survey with 
automated branching and error checking. Results revealed that item nonresponse in all 
three web modes exceeded mail and email modes, with the highest item nonresponse 
occurring in the web mode with automated branching and error checking (the email 
response rate was too low to provide a valid comparison). They concluded that the use of 
web surveys with multiple pages compared to a single page will likely increase the 
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number partially completed web surveys, especially if error checking is used (Smee & 
Brennan, 2000). Reducing item nonresponse is best achieved by improving item design 
rather than by forcing respondents to fix their responses (Dillman, 2000; Smee & 

Brennan, 2000). 

Item Completion Error 

Item completion error is a measure of the number of errors on a survey 
questionnaire and happens when a respondent provides an invalid response (e.g., a 
response where one is not expected, a response that cannot be coded, illegible 
handwriting) (Pettit, 2002). Pettit (2002), using the term “response error” instead of item 
completion error, used the following operational definition: 

The total number of items for which (1) two or more options were selected, (2) it 
was not obvious which option was finally selected, (3) the response was illegible, 
(4) the response was inappropriate or impossible, or (5) a demographic 
combination was unlikely, (p. 52) 

Sometimes this is the result of a branching or data entry instruction that was not 
understood or followed correctly (Dillman, 2000). General suggestions to reduce item 
completion errors in surveys, regardless of mode include providing concise instructions, 
using appropriately designed contingency items, and field-testing the survey questionnaire 
(Babbie, 2007; Dillman, 2000; Fraenkel & Wallen, 2006). Suggestions to reduce item 
completion errors in web surveys, in particular, are similar to those used to reduce item 
nonresponse in surveys: automated branching and error checking (Schaefer & Dillman, 
1998; Tourangeau, 2004). 

The literature is inconsistent, with some studies reporting no difference in item 
completion errors among mail and web surveys (Pealer, 1999; Smee & Brennan, 2000), 
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and others reporting fewer item completion errors in web compared to mail surveys 
(Weible & Wallace, 1998; Pettit, 2002). Smee and Brennan (2000) found no differences 
in their comparison of mail, email, and web modes. Kiesler and Sproull’s (1986) health- 
related survey among 151 university students and faculty/staff reported fewer item 
completion errors in the email as compared to the mail survey. Of 53 items containing 
response errors, 5.3% were made by mail respondents and 0.0% was made by email 
respondents (Kiesler & Sproull, 1986). 

In the current study, item completion error was examined among items in which a 
respondent provided an answer where one was not expected based on the respondent’s 
response to a prior contingency-type item. This definition most closely relates to the 
fourth part of Pettit’s (2002) operational definition, referring to responses that are 
inappropriate or impossible. The first part of Pettit’s (2002) operational definition, where 
two or more responses are selected, was examined in the current study using the “multiple 
response use” response quality variable. Other ways to measure item completion error 
were beyond the scope of the current study. 

Response Length 

Response length is a measure of the number of words in a respondent’s response 
to an open-ended item. It is generally assumed that a longer response written to an open- 
ended item is indicative of better response quality because respondents provide more data 
(Schaefer & Dillman, 1998). Email surveys have been shown to produce longer responses 
to open-ended items than mail surveys (Paolo et ah, 2000). Several studies have observed 
that web surveys surpass paper/mail surveys in producing higher quality, more 
sophisticated, and longer responses to open-ended items (Kiesler & Sproull, 1986; 
MacElroy et ah, 2002; Nicholls et ah, 1997; Schaefer & Dillman, 1998; Wu, 1997). In a 



59 




web and mail survey involving mode self-selection (n = 1,228), a mean of 48.2 words 
was revealed for the web mode compared to a mean of 32.0 words for the mail mode for a 
“Use this space to add some final thoughts” open-ended item (MacElroy et ah, 2002). 
Yea-Saying 

Yea-saying is the tendency of a respondent to agree with item statements or 
questions independent of the item’s content (Greenleaf, 1992) as evidenced by a response 
that is located near or at the affirmative anchor of a list of Likert-type response options 
(Kiesler & Sproull, 1986). A study of yea-saying revealed no differences between 
telephone, mail and web surveys (Weijters et ah, 2004). In a study conducted by Kiesler 
and Sproull (1986) comparing paper and electronic surveys, yea-saying was a measure of 
respondents’ use of the affirmative side of a seven-point attitudinal scale and response 
extremity was a measure of their tendency to avoid extreme responses on the same scale 
(e.g., prefer the scale’s midpoint). Yea-saying and response extremity were measured 
among 151 randomly-selected university students and faculty/staff using five forced- 
choice items. The items solicited participants’ attitudes on health-related topics using a 
seven-point Likert response set where “Agree” equals one and “Disagree” equals seven. 
ANOVAs revealed no differences in yea-saying and no differences in response extremity 
by survey mode (Kiesler & Sproull, 1986). 

The current study sought to expand the measurement of yea-saying in the 
literature by analyzing a dichotomous yes-no response scale instead of the seven-point 
Likert scale used by Kiesler and Sproull (1986). Yea-saying was thus examined by 
counting the number of yes responses and analyzing differences by survey mode and 
response wave using ANOVAs. 
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Response Extremity 

Response extremity occurs when a respondent favors (or avoids) the extreme 
anchors (e.g., “strongly agree” or “strongly disagree” in a Likert scale) of a response scale 
(Greenleaf, 1992; Kiesler & Sproull, 1986; Naemi, 2006; Pettit, 2002). There is 
considerable variation in the way response extremity is measured in the literature (Naemi, 
2006). Some studies measure response extremity by counting use of both extreme anchors 
in Likert-type response sets (e.g., “definitely disagree” and “definitely agree”) (Berg & 
Collier, 1953; Booth-Kewley et ah, 1992; Greenleaf, 1992; Pettit, 2002; Soueif, 1958; 
Sproull, 1986). For this type of measurement, Pettit (2002) provides the following 
operational definition: “the percentage of answered items for which either strongly agree 
or strongly disagree was selected” (p. 52). Some studies measure response extremity by 
counting use of only the most extreme positive response option (Brengelmann, 1960), 
while others use standard deviations (Hamilton, 1968). 

A study using a community satisfaction questionnaire in telephone and mail 
survey modes revealed that telephone respondents used extreme responses more than mail 
respondents, but telephone respondents following the interview with a paper survey in 
hand preferred the middle response options (Tamai & Dillman, 1992). They concluded 
that middle categories were favored in surveys using a visual fonnat (Tamai & Dillman, 
1992). Dillman et al. (2001) supported this conclusion in a study among 8,999 
participants in a mixed-mode (e.g., telephone, IVR, mail, and web) survey about long 
distance telephone services. They analyzed items containing five-point Likert-type 
response options (e.g., “Not at all satisfied” to “Extremely satisfied”), although how they 
measured response extremity is not clear. They observed that telephone respondents used 
extreme responses more than web respondents, but there was no difference among mail 
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and web respondents (Dillman et al., 2001). Helgeson and Ursic (1989) observed no 
differences between paper and electronic surveys in a sample of undergraduate students. 
Pettit (2002) observed no differences between web and paper modes in a comparison of 
personality questionnaires. Booth-Kewley et al.’s (1992) study among 246 Navy recruits 
also reported no differences between a computer survey with backtracking (e.g., 
respondents could return to previous pages and change their responses), a computer 
survey with no backtracking, and a paper survey. In contrast, Sproull’s (1986) survey 
among 60 business professionals in a Fortune 500 company reported more extreme 
responding in email than in paper surveys. This study examined 48 hypothetical task 
items with response anchors scaled from zero to 10 where zero and 10 were coded as 
extreme responses (Sproull, 1986). 

The current study sought to expand the measurement of response extremity to 
include a four-point percentage-based interval scale using the most extreme positive 
response as suggested by Brengehnann (1960). Response extremity was thus measured by 
counting the number of “76-100%” responses in items containing percent scaled 
responses (e.g., 1-25%, 26-50%, 51-75%, and 76-100%) and then analyzed by survey 
mode and response wave using ANOVAs. 

Response Equivalency 

Response equivalency is a measure of differences in the mean scores among items 
or collapsed item groups having ordinal, interval or ratio level data. Comparing these 
mean scores across survey modes permits a researcher to analyze whether respondents 
answer differently by mode. Researchers suggest that responses across surveys modes in 
general are not equivalent (Dillman, 2000; Sudman et al., 1996) due to differential effects 
on responses resulting from each mode’s degree of impersonality, perception of 
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legitimacy, and level of cognitive burden imposed on the respondent (Tourangeau et al., 

2000, p. 20). 

One study comparing paper and email observed a difference in response 
equivalency (Miller et al., 1996), but most mail and electronic survey studies reported no 
differences (Bachmann et al., 1996; Booth-Kewley et al., 1992; Hayslett and Wildemuth, 
2004; Helgeson & Ursic, 1989; Mehta & Sividas, 1995; Pettit, 2002). In a study 
comparing personality questionnaires in web and paper modes, Pettit (2002) observed no 
differences in mean scale scores and no differences in internal consistency by survey 
mode. Helgesen and Ursic (1989) examined the equivalency of decision processes among 
126 randomly assigned undergraduate business students via electronic and paper surveys. 
They observed no statistically significant differences in response equivalency and no item 
ordering effects by survey mode. They noted, however, that electronic responses were 
more stable than paper responses when the extremity of scalar anchors changed 
(Helgeson & Ursic, 1989). Booth-Kewley et al.’s (1992) study among 246 Navy recruits 
measured response equivalency by comparing the mean scores of items containing five- 
point Likert response sets (e.g., 1 = strongly agree, 5 = strongly disagree). They reported 
no difference in response equivalency between paper, computer with backtracking, and 
computer without backtracking surveys (Booth-Kewley et al., 1992). 

Multiple Response Use and Anecdotal Comments 

Hayslett and Wildemuth (2004) noted in their study among 300 academic 
references librarians that the use of check boxes in the mail survey versus radio buttons in 
the web survey produced a data entry problem in an item about primary work 
responsibilities. Only one response was expected for this item. In the web survey, the use 
of radio buttons prevented respondents from selecting more than one response option; 
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however, some mail survey respondents checked two or more responses on their mail 
surveys. Thus, researchers applied a rule to the mail survey responses to select which one 
among the multiple responses would be retained and which would be discarded (Hayslett 
& Wildemuth, 2004). No analysis was conducted on the discarded responses. Pettit’s 
(2002) study comparing personality scales by paper and web modes provided the 
following operational definition of multiple response use: “The total number of items for 
which. . .two or more options were selected” (p. 52). Pettit (2002) analyzed multiple 
response use as one of five types of errors measured collectively as response errors (item 
completion errors), thus no findings for multiple response use alone were reported. 

The current study sought to examine this phenomenon in greater detail. Similar to 
Hayslett and Wildemuth (2004), rules were applied in the current study to select which 
one among multiple responses to retain in instances where only one response was 
expected. A variable was then created to count the number of responses that would have 
otherwise been discarded. Since radio buttons used in the web survey prevented 
respondents from providing multiple responses where they weren’t supposed to, this 
analysis could only be examined in the current study in the mail survey mode by response 
wave. 

Hayslett and Wildemuth (2004) noted another problem in their study, the use of 
anecdotal comments by mail respondents but not by web respondents. They explained 
that some mail respondents wrote notes on their paper surveys regarding items they 
thought were ambiguous or didn’t understand, to describe how they interpreted an item, 
and to clarify their responses. The web survey design, however, prevented web 
respondents from writing such comments. They suggested adding text boxes at key points 
throughout the web survey to capture anecdotal comments, but noted that web 
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respondents would still not be able to draw lines, circle portions of items, or write 
nontextual annotations as mail respondents could (Hayslett & Wildemuth, 2004). 

The current study sought to examine this phenomenon in greater detail as well. A 
variable was created to record all textual anecdotal comments, and then analyzed for 
response length and pronoun use. Since web respondents in the current study were 
prevented by the web survey interface from writing anecdotal comments, this variable 
could only be analyzed in the mail survey mode by response wave. 
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CHAPTER III 



METHODOLOGY 

Chapter 3 presents the methodology used to answer the research questions in this 
study. It is arranged into the following nine sections: research design, using an existing 
dataset, research questions, variables, instrumentation, data collection, population and 
sampling, statistical analysis, and analysis of research questions. 

Research Design 

This study employed a nonexperimental, quantitative, cross-sectional survey 
research design to examine demographic characteristics and response quality among mail 
and web survey modes at two time periods. The sample was nonrandomly selected and 
nonrandomly assigned to groups. Respondents self-selected the survey administration 
mode and self-administered the survey. There was no control group. Respondents were 
divided into one of two survey modes (Mail or Web) and one of two data collection 
cycles (Wave 1 or Wave 2) comprising four groups (Maill, Mail2, Webl, and Web2) 
with unequal sample sizes based on the survey mode each selected and the time period in 
which each responded. To distinguish each of the respondent group variables, the naming 
convention in Table 1 was employed. All respondents were instructed to complete their 
surveys based on the caseload of 6th- 12th grade students they had during the 2004-05 
academic school year. 

Maill and Webl respondents responded during the Spring 2005 data collection 
period, which occurred April through July 2005 at the end of the 2004-05 academic year. 
Maill respondents submitted mail surveys and Webl respondents submitted web surveys 
accessed via the Internet. Mail2 and Web2 respondents responded during the Fall 2005 
follow-up data collection period, which occurred September through December 2005 at 
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the beginning of the 2005-06 academic year. Mail2 respondents submitted mail surveys 
and Web2 respondents submitted web surveys accessed via the Internet. The fact that data 
collection occurred in two distinct cycles during two different school years permitted an 
examination of possible mode differences among early and late respondents. Table 2 
shows the respondent group comparison combinations used in the analyses. 

Table 1 

Naming Convention for Survey Mode and Response Wave Response Condition Variables 



Respondent 

Condition 


Variable 

Name 


Description 


Survey Mode 


Mail 


All mail surveys returned during the Wave 1 and Wave 2 
combined data collection cycles 




Web 


All web surveys returned during the Wave 1 and Wave 2 
combined data collection cycles 


Response Wave 


Wave 1 


All mail and web surveys combined that were returned during 
the Wave 1 data collection cycle 




Wave 2 


All mail and web surveys combined that were returned during 
the Wave 2 data collection cycle 


Survey Mode x 


Maill 


Mail surveys returned during Wave 1 


Response Wave 


Mail2 


Mail surveys returned during Wave 2 




Webl 


Web surveys returned during Wave 1 




Web2 


Web surveys returned during Wave 2 



Table 2 




Respondent Group Comparison Combinations 


Comparison Group 


Design 


Survey Mode 


Mail vs. Web 


Response Wave 


Wave 1 vs. Wave 2 


Mail by Response Wave 


Maill vs. Mail2 



Web by Response Wave Webl vs. Web2 

Survey Mode at Wave 1 Maill vs. Webl 

Survey Mode at Wave 2 Mail2 vs. Web2 

Note. While interaction effects were analyzed to 
determine whether to examine main effects or simple 
effects, there were no research questions that specifically 
analyzed Maill vs. Web2 or Mail2 vs. Webl conditions in 
this study. 
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Using an Existing Dataset 



The current study used a dataset derived from a larger study (Kohler, Applegate, 
Bradley, Cai, & Bradshaw, 2007) that examined school counselor work settings, 
caseloads, career assessment and planning activities, and recruitment and support 
strategies for students with and without disabilities in nontraditional occupations. The 
larger study was approved by WMU’s Human Subjects Institutional Review Board 
(HSIRB) in two parts: “Creating IT Career Pathways through High School Career and 
Technical Education Programs, Part 2” (HSIRB Project #05-03-20) and “Creating Career 
Connections for Students with Disabilities through Career and Technical Education 
Programs, Part 2” (HSIRB Project #05-03-21). Protocols for analyses of data for these 
two studies were approved by the WMU HSIRB as “Creating IT Career Pathways 
through High School Career and Technical Education Programs, Part 1” (HSIRB Project 
#03-05-23) and “Creating Career Connections for Students with Disabilities: A 
Longitudinal Study of Their Enrollment and Outcomes of Career and Technical 
Education, Part 1” (HSIRB Project #03-08-05), and by the University of Illinois’ HSIRB 
as Cases #03232 and #03233, respectively. The University of Illinois’ HSIRB, which also 
serves as the Illinois State Board of Education’s HSIRB, approved release of participants’ 
names to Kohler for the larger study. 

The “Creating IT Career Pathways” and “Creating Career Connections” studies 
comprised a research partnership between WMU, the Illinois State Board of Education 
(ISBE), and Research-to-Practice Teams of business and practitioner stakeholders. The 
“Creating IT Career Pathways” study, funded by the National Science Foundation (grant 
#0306092), investigated factors influencing the enrollment, program concentration, and 
employment experiences of high school females in Career and Technical Education- 
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Information Technology (CTE-IT) programs in Illinois. The “Creating Career 
Connections” study, funded by the U.S. Department of Education Office of Special 
Education Programs (grant #H324C030014), explored factors influencing the enrollment, 
program concentration, and employment experiences of high school students with 
disabilities in Career and Technical Education (CTE) programs in Illinois. Both projects 
were based at WMU’s Career Connections Resource Center and implemented 
collaboratively with ISBE. 

In response to federal accountability mandates, each year ISBE collects data 
regarding student enrollment and perfonnance in CTE programs throughout the state, as 
well as postsecondary education and employment data. Using these data collected by 
ISBE, the “Creating IT Career Pathways” study explored preparation for IT careers in the 
following areas: (a) school counselor and teacher support strategies to recruit and retain 
students in CTE-IT programs; (b) student and school characteristics that influence CTE- 
IT program enrollment, school-directed work experience, and program concentration; and 
(c) the influence of student and environmental characteristics and support strategies on 
post-secondary enrollment in IT education programs and employment. Using the same 
participants and data, the “Creating Career Connections” study explored the enrollment 
and outcomes of students with disabilities as compared to their peers without disabilities 
in CTE programs in Illinois across four dimensions: (a) career assessment; (b) 
enrollment; (c) outcomes; and (d) policy influences. In both projects, experiences of 
students with disabilities were compared with those of students without disabilities; 
experiences of students from various ethnicities and of different genders were also 
investigated. 
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As part of the larger study, surveys were conducted among CTE instructors and 
school guidance counselors working with students in grades 6 through 12 in public 
schools and CTE centers in Illinois during the 2004-2005 academic year. CTE instructors 
in the sample received a 29-item survey questionnaire in web and mail formats, and 
counselors received a 48-item survey questionnaire in web and mail formats. Participants 
were permitted the option to respond to their survey via mail or the web. The larger 
studies provided a means to examine the methodological issues of survey research by 
comparing web and mail survey modes. The data set from the larger studies was used in 
the current study to compare demographic characteristics and response quality among 
survey modes. Participants in the CTE survey portion of the study were excluded from 
the current study because the demographics of computer and technology instructors in 
that sample could potentially bias the results (Kaminer, 1997). Thus, the current study 
included only participants involved in the 48-item school counselor survey. 

Given that the current study used an existing dataset from the larger study, all 
procedures leading to the creation of this dataset were conducted before the current study 
was initiated (e.g., instrument design, field testing, sampling, survey dissemination, data 
collection, and data entry); however, they are described in this study in detail so that 
analyses and results can be examined within the context of the entire study. The dataset 
used by the researcher in the current study was stripped of all identifying information 
(e.g., participant names, school names). All protocols in the current study were approved 
by Western Michigan University’s Human Subject’s Institutional Review Board under the 
“exempt” category of review on April 28, 2006 as HSIRB Project Number 06-04-24 (see 
Appendix A). 
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Research Questions 



The following research questions were examined in this study: 

1 . Does a difference exist in demographic characteristics by survey mode or 
response wave? 

a) Does a difference exist in gender by survey mode or response wave? 

b) Does a difference exist in ethnicity by survey mode or response wave? 

c) Does a difference exist in age by survey mode or response wave? 

d) Does a difference exist in professional employment by survey mode or 
response wave? 

2. Does a difference exist in response quality by survey mode or response wave? 

a) Does a difference exist in pronoun use by survey mode or response 
wave? 

b) Does a difference exist in pronoun use in anecdotal comments by 
response wave? 

c) Does a difference exist in item nonresponse by survey mode or 
response wave? 

d) Does a difference exist in response extremity by survey mode or 
response wave? 

e) Does a difference exist in yea-saying by survey mode or response 
wave? 

f) Does a difference exist in item completion errors by survey mode or 
response wave? 

g) Does a difference exist in response length by survey mode or response 
wave? 
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h) Does a difference exist in the response length of anecdotal comments 
by response wave? 

i) Does a difference exist in response equivalency by survey mode or 
response wave? 

j) Does a difference exist in multiple response use by response wave? 

3. Do demographic characteristics and response quality predict survey mode or 

response wave? 

a) Do demographic characteristics and response quality predict survey 
mode? 

b) Do demographic characteristics and response quality predict response 
wave? 

Variables 

This study examined demographic and response quality variables by survey mode 
and response wave. Categorical demographic variables were gender with two levels 
(male, female) and ethnicity with seven levels (Black or African American, American 
Indian or Alaskan Native, Asian, White or Caucasian, Latino or Hispanic, Native 
Hawaiian or Other Pacific Islander, and Biracial or Multiracial). Continuous demographic 
variables were age and professional employment, each measured in years. Continuous 
response quality variables examined for all levels of survey mode and response wave 
were: pronoun use, item nonresponse, response extremity, yea-saying, item completion 
errors, response length, and response equivalency . Two additional continuous response 
quality variables, anecdotal and multiple response use, were analyzed only for Maill and 
Mail2 conditions because the web-based interface prevented them from occurring in the 
web survey. All response quality variables except response equivalency were count-type 
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variables in which values reflected a count of the total number of instances. Response 
equivalency, in contract, was measured as a mean. Response condition variables were 
survey mode (mail and web) and response wave (one and two). See Table 3 for variables 
and their constitutive and operational definitions as they apply to the current study. 

Instrumentation 

The survey instrument was a self-administered questionnaire taking approximately 
30 minutes to complete and containing 48 items. Two parallel versions of the 
questionnaire were developed: a paper fonnat that was mailed to respondents (see 
Appendix D) and a web format which respondents accessed via the Internet (see 
Appendix E). Efforts were made to keep the two versions as similar as possible to 
facilitate a comparison of web and mail survey modes. Participants received no incentive 
to participate, but were asked to provide their email addresses if they desired a report of 
the study’s findings. 

Content, Validity, and Reliability 

Questionnaire items were developed from the following sources: 1) career 
counseling and student support strategies from the literature; 2) competencies suggested 
by the American School Counselor Association; and 3) data provided by the National 
Alliance for Partnerships in Equity (Kohler et al., 2007). Content sections were arranged 
as follows: 1) School counselor work setting and caseload (items 1-13); 2) Career 
assessment activities with students on caseload (items 14-28); 3) Career planning 
activities with students on caseload (items 29-31); 4) Recruitment and support for 
students in nontraditional occupations (items 32-40); and 5) Demographic information 
and open-ended comments (items 41-18). The survey’s content validity was examined by 
content experts to detennine the extent to which concepts in the survey correctly 
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Definitions and Measurement Levels for All Study Variables 
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noun phrases including their contractions (e.g., I’d, I’ve, I’ll, I’m, we’d, we’ve, we’ll, we’d) 

used by respondents in response to questions 45-46 and 48. 

Table Continued 
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to determine if they are equal. and “NA” categories were removed from analyses because they did not have the 

same intervals as the remaining categories. The mean score was obtained by 
adding the score for each sub-item into a total score, and then dividing by the total 
number of sub-items that each respondent answered. 




represented the scope of concepts that should be included. The survey was approved for 
its intended use. 

Instrument reliability, the degree to which an instrument produces consistent 
scores, was not examined in this study for several reasons. First, the design of the larger 
study (Kohler et al., 2007) was exploratory in nature, with the purpose of conducting a 
situational analysis of school counselors’ actual versus intended (e.g., within the scope of 
their job) behaviors at a specific point in time and in a specific context. Second, 
researchers accepted the assumption that if respondents chose to participate, they did so 
with the purpose of providing truthful (e.g., valid) and consistent (e.g., reliable) 
responses. Third, test-retest and equivalent forms reliability analyses were not possible or 
practical in the larger study, both requiring two administrations of the instrument to the 
same group. The time and resource costs expended for such analyses exceeded the benefit 
that would be contributed to the study’s exploratory purpose. Finally, considering item 
content and structure, too few items were available to provide a meaningful internal 
consistency analysis. An internal consistency analysis is used to examine how well 
different items complement each other in their ability to measure the same construct or 
dimension (Fink, 2006; Fraenkel & Wallen, 2006). The survey in the current study did 
not use multiple items to measure one dimension; each item was instead designed to 
measure a different dimension. 

Item Construction 

The 48-item instrument contained the following kinds of items: 34 forced-choice 
questions with categorical responses, six short-answer questions, four questions with both 
short-answer and forced-choice responses, and four open-ended questions. Many items 
were divided into multiple sub-items, so among the 48 main items, there was a total of 
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253 sub-items. See Figure 1 for an example of an item that combines both short answer 
and forced-choice responses. Though counted as one item, this item actually contained 10 
sub-items, one for each open-ended response option and one for each row of forced- 
choice response options. Forced-choice response options for this type of item were coded 
as 1 = None, 2 = 1-25%, 3 = 26-50%, 4 = 51-75%, and 5 = Over 75%. 



Figure 1. Sample Item Combining Short Answer and Forced-Choice Response Options 




Stems and response categories were designed with careful attention to issues of 
tenninology and consistency. Key terms were defined to reduce confusion, and then 
underlined, bolded, or italicized for clarity where they appeared in a stem or response. For 
example, “students with disabilities” was defined as follows: “Students with disabilities 
are students who receive special education services.” Then, the tenn “with” in “students 
with disabilities” and the tenn “without” in “students without disabilities” were bolded to 
distinguish them. These comparison items appeared several times throughout the survey, 
so in each instance, “students with disabilities” preceded “students without disabilities” 
for consistency. Care was taken to use identical main stems when asking questions 
designed to compare concepts. In six different items, there appeared a single main stem, 
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followed by two dichotomous sub-stems (with and without disabilities), followed by a set 
of forced-choice response options. See Figure 2 for an example of this item type. Though 
considered one item, this item was actually divided into two sub-items, one for the 
“students with disabilities” response and one for the “students without disabilities” 
response. Response options for this item were coded as 1 = No One, 2 = 1 Do, 3 = Other 
Person, and 4 = NA. 

Figure 2. Sample Item with Multiple Sub-Items 

21. In your school, who has primary responsibility for implementing career 
assessment(s) for the students on your caseload? 





No one 


I do 


Other 

person 


NA 


Students with disabilities 


□ 


□ 


□ 


□ 


Students without disabilities 


□ 


□ 


□ 


□ 



Some items were designed such that multiple questions were asked 
simultaneously using an “item-in-a-series” fonnat (Dillman, 2000). For these items, 
questions were asked that used the same response categories, but distinguished responses 
between two concepts. See Figure 3 for an example of this item type. Though considered 
one item, this item was actually divided into six sub-items. Response options were for 
this type of item were coded as 1 = Yes and 2 = No. For some analyses, 2 = No was 
recoded as 0 = No. 

Survey Modes 

Two parallel versions of the questionnaire were developed, each containing 48 
items. The mail version contained 16 separate printed pages, including the front and back 
cover which contained no items (see Appendix D) and the web version contained 10 
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separate web pages (see Appendix E). The mail version was created using Microsoft 
Word. The web version was created using online survey software provided by Enterprise 
Survey Software Systems, Qualtrics Labs, via their web site at http://www.SurveyZ.com. 
Figure 3. Sample Item-in-a-Series Format 



For questions 33-40, please indicate (1) whether you do the activity and (2) whether you 
consider the activity to be within the scope of your job. 



36. Conduct professional development 
regarding: 

Gender and/or ethnic bias 
NT careers and/or programs 
Sexual harassment prevention 



Do the activity 
Yes No 


Within scope of job 
Yes No 


□ 


□ 


□ 


□ 


□ 


□ 


□ 


□ 


□ 


□ 


□ 


□ 



This software service was selected for features including convenience and ease of 
use, cost, web hosting and site security, the ability to handle multiple respondents 
simultaneously, and the ability to export data to Microsoft Excel and SPSS fonnats for 
analysis. Web survey responses were automatically entered into a secured database hosted 
by the server to be downloaded later by researchers. 

Given that the mail version was each respondent’s first contact with the survey 
(the web link was printed on the cover of the mail version), attention was devoted to 
preparing an official and professional-looking mail survey using some of Dillman’s 
(2000) and others’ suggestions. The front and back covers identified government 
sponsorships (as suggested by Dillman, 2000; Dillman et al., 1996; Heberlein & 
Baumgartner, 1978) as well as Western Michigan University’s Career Connections 
Research Center and the researchers’ names and contact information (Dillman, 2000) (see 
Figures 4 and 5). 
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The opening page of the web version contained no logos, but identified the WMU 
Career Connections Research Center and included telephone and email contacts (see 
Figure 6). The page was designed so the university name and welcome messages 
comprised the first image seen by respondents, then they would scroll down to begin 
section one. Note in Figure 6 the “3% Complete” message. Each page contained a colored 
progress bar above this message; however, the bars do not appear in the web page 
screenshots. 

Figure 4. Front Cover of Mail Survey 



■ , ni 

I I I 



Career Connections Research Center 

Western Michigan University 

Phone: 269-387-6181 
Fax: 269-387-2826 



School Counselor Survey 

Spring 2005 



You may take the survey on-line at: 

httpiftwww.surveyz com,TakeSurvev?id=23228 



A letter of support for the study from the Illinois State Board of Education, written 
on ISBE letterhead, was included in the principal’s packet; cover letters to principals and 
participants were written on university letterhead (see Appendix B and Appendix C) 
(Dillman, 2000). The survey was designed as a booklet using 8.50 x 14 inch (legal size) 
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paper, folded in half with two staples in the spine. This booklet format is considered 
acceptable for multi-page surveys with the added benefit that when folded lengthwise fits 



nicely into a standard business stationery envelope (Dillman, 2000). 



Figure 5. Back Cover of Mail Survey 
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Web survey formats have been classified into two main types: static (scrollable) 
and dynamic (interactive) (Dillman, 2000; Tourangeau, 2004). Research is inconclusive 
regarding which format is better, noting that the static format is better for some tasks 
while the dynamic is better for others (Dillman, 2000; Norman, Friedman, Norman, & 
Stevenson, 2001). Dillman (2000) advocates the static approach, which displays the entire 
survey on a single web page, contains no automated features, permits respondents to 
scroll up and down the page, and allows respondents to change their answers. The 
dynamic format uses all or most of the available automated features. Clicking the 
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“Submit” button in either web survey format would be equivalent to mailing a completed 
paper survey. The web survey in the current study used a combination of static and 
dynamic formats. A “Continue” button at the bottom of the web survey’s ninth page 
followed by a final page (“Thank Your For Participating in Our Survey! Your responses 
have been saved and recorded with ID xxxxx.”) was equivalent to clicking the “Submit” 
button suggested by Dillman (2000). 

Figure 6. Screenshot of Web Survey Opening Page 
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Special attention was devoted to keeping the two versions as similar as possible. 
Both versions had exactly the same items in the same order; however, some differences 
occurred due to the nature of the software used to created the web version. Where check 
boxes and lines were used for forced-choice and open-ended response options in the mail 
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version, check boxes (for “Check all that apply” items), radio buttons (for “Check only 
one” items) and text boxes were used in the web version. Where written instructions were 
provided in the mail version for questions five and 15 to indicate branching items, 
automated branching was used in the web version to avoid making web respondents click 
and scroll through several pages of items that didn’t apply. Thus, these two branching 
scenarios could be analyzed only by response wave in the mail version. Automated 
branching was not used in the web version for questions 10 and 14, thereby permitting an 
analysis of these items by survey mode and by response wave. No error-checking or pop- 
up features were used in the web version, and participants in both modes were pennitted 
to skip items. 

The mail version pennitted participants to move forward and backward through 
the instrument, answer items out of order, and ignore instructions. The web version 
pennitted respondents to move forward and backward and answer items out of order, but 
only within a web page. Once a web respondent had moved to a subsequent web page, 
they were prevented from returning to a previous page. There were also some item 
numbering and formatting differences because the online software provided less 
flexibility in designing matrix-type items (see Figure 7). 

Two key differences in the handling of survey responses by mode were the use of 
“select only one” items and anecdotal comments. For “select only one” items, web 
respondents were prevented from selecting multiple items; however, mail respondents 
could select multiple responses regardless of the instruction not to do so. Additionally, 
some mail respondents took the opportunity to make anecdotal comments in the margins 
of their survey items, often to clarify a response; however, the web respondents could 
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only write responses in text boxes. Thus, an analysis of the use of multiple responses and 

anecdotal comments could only be conducted in the mail survey by response wave. 

Figure 7. Sample Formatting Differences Between Identical Items in the Mail (Top) Versus the 
Web (Bottom) Survey 

17. In general, what percentage of the students on your caseload participate in career 
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16. In general, what percentage of the students on your caseload participate in career assessment(s) 
some time during their secondary education? 
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Field Testing 

Both questionnaire formats were field tested among with a volunteer group 
comprised of 10 Michigan school counselors who were identified by Western Michigan 
University’s Counseling Education and Counseling Psychology faculty. Participants each 
received a $10.00 gift certificate to a major discount store. 

Data Collection 

Since participants’ email addresses were not available, it was necessary to provide 
initial contact with participants via mail. Additionally, since there were likely to be errors 
in the ISBE Teacher Service Record report which contained the names of potential 
participants, the decision was made to confirm the list of names with school building 
principals before surveys were distributed to participants. 
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The first data collection cycle, “Wave 1,” was conducted April through July 2005 
and the second data collection cycle, “Wave 2,” was conducted September through 
December 2005. In late April 2005, a packet was mailed to the principal of each school 
building where a participant from the target population worked. Principal packets 
contained an introductory letter, letter of support from the Illinois State Board of 
Education, Principal’s Distribution List, and separate survey packets for each participant 
in their school (see Appendix B). The introductory letter noted WMU’s HSIRB approval 
(for the larger study) and described the study, how participants were identified, 
confidentiality, the option for participants to self-select the survey mode, and instructions 
for distributing surveys to their school counselors. 

The Principal’s Distribution List noted the name and job title of each participant 
in the school invited to participate. Principals were instructed to make corrections or add 
new names to the list as necessary to reflect all of the guidance counselors in the school. 
Principals were then instructed to return the corrected Principal’s Distribution List in an 
enclosed postage-paid return envelope so researchers could update their participant list. A 
survey packet for each counselor in the school was provided that included an introductory 
letter; instructions to voluntarily complete the survey; notice of confidentiality, informed 
consent, risks, and IRB approvals; approximate amount of time to complete the survey 
(about 30 minutes); a paper version of the survey questionnaire with the Internet address 
of the web version printed on the cover; and a postage paid return envelope (see 
Appendix C). All respondents received identical information and instructions and the web 
version of the survey was activated the same day that survey packets were delivered to 
school principals. 
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As Principal’s Distribution Lists were received, researchers updated their 
participant list and removed the principals’ names from the list of principals to receive 
follow-up notices. About two weeks after the initial mailing, a reminder postcard was 
delivered to each principal who had not returned their list. About two weeks after that, 
principals who had still not returned their lists were contacted by phone by a member of 
the research team to inquire about the status of their list and request that the list be 
returned. 

As surveys were returned from respondents, the respondent’s name was removed 
from the list of nonrespondents, and then each survey was assigned a unique code to 
protect confidentiality. Mail survey data was then manually entered into a database. Web 
survey responses were downloaded by researchers from the web survey host’s server into 
Excel files. After assigning the unique codes, each respondent’s name was permanently 
deleted from the data file. Using Dillman’s (2000) suggestions to improve response rates, 
the initial mailing was followed with four first-class reminders. Following the initial 
delivery of surveys to respondents in April 2005, first reminder postcards were delivered 
in May, second reminder letters and replacement surveys were delivered in early June, 
and third reminder postcards were delivered in mid- June. Fourth and final reminder 
letters and replacement surveys were delivered to nonrespondents in October 2005, 
producing the “Wave 2” portion of the data collection cycle. This is important given that 
Wave 2 respondents completed surveys during the 2005-2006 academic year, but items 
referred to their caseload of students during the 2004-05 academic year. 

Although studies suggest that pre-notification letters increase response rate 
(Dillman, 2000), sending pre-notification letters was not possible in this study due to time 
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constraints. The survey packet each respondent received from their school principal was 
the respondent’s first notification of the survey. 

Population and Sampling 

According to Schonlau et al. (2002), the most important question facing a survey 
researcher in whether the study will use a sample to make inferences about a larger 
population. If inferences are to be made, then a probability (e.g., random) sample is 
needed, but if not, a nonprobability sample may suffice (p. 74). Given the exploratory 
nature of the larger study from which the current study is derived, the decision was made 
that a random sample was not necessary. 

School counselors in Illinois public schools counseling students in grades 6-12 
comprised the target population (the population of interest). The names of 3,019 potential 
participants in 1,502 Illinois public schools were identified via an Illinois State Board of 
Education (ISBE) Teacher Service Record report. Using information from the ISBE 
website (http://www.isbe.net/), researchers identified school principals at each site where 
a school counselor worked. Building and staffing changes occurred in the time between 
identifying the target population (October 2004) and delivering surveys to participants 
(April 2005). Therefore, survey packets were first delivered to principals to confirm their 
counseling staff identities against the names of counselors from the Teacher Service 
Record report. Principals were instructed to note changes in the list including counselors 
who were on the list and did not receive a survey (e.g., extended illness, maternity leave, 
retirement, no longer employed, incorrect address, transferred to another school), 
counselors who received a survey and were not on the list (e.g., new person replacing a 
counselor in an existing position), counselors who were on the list and received a survey 
(including any changes in their names or position titles), special staffing changes (e.g., a 
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position was eliminated or not filled), and whether more surveys were needed (e.g., 
adding the names of new counselors in new positions). Principals then delivered a survey 
packet to each of their counselors, and returned the principal-verified list to researchers. 
All participants ( n = 2,880) on this principal-verified list comprised the accessible 
population (sampling frame) of participants who were available to participate in the 
survey. The sampling method is considered a convenience sample (Fraenkel & Wallen, 
2006; Schonlau et ah, 2002). 

The participants in the accessible population had a mean of 19.28 years of 
professional employment (n = 2,730, SD = 10.57) and worked in schools with a mean 
enrollment of 1,364.55 students (n = 2,710, SD = 964.68). Among them, 94.46% (n = 
2,579) had a masters or doctorate degree, and 93.30% (n = 2,547) worked full time. See 
Table 4 for characteristics of the accessible population. A chi-square goodness-of-fit 
analysis was conducted to compare the respondents to the accessible population to 
determine whether the observed proportions in the sample were statistically different 
from theoretically expected proportions in the population (Glass & Hopkins, 1996) (see 
Chapter 4). 

Statistical Analysis 

Chi-square (x ) Analysis 

The chi-square goodness-of-fit test is a statistic that determines whether the 
observed proportions in a sample with two or more groups are statistically different from 
theoretically expected proportions in the population (Glass & Hopkins, 1996). This 
statistic was used to compare the sample (n = 880) to the population (n = 2,880). 
Goodness-of-fit analyses were also conducted for the purpose of model checking. 
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Categorical variables were first analyzed with a three-way chi-square: 2 (e.g., 
gender: male, female) x 2 (survey mode: mail, web) x 2 (response wave: one, two) where 
survey mode and response wave fonned the rows and columns respectively. This analysis 
permits examination of the association between survey mode and response wave while 
controlling for the effect of each level of the demographic variable. 

Table 4 



Demographics of Accessible Population (N = 2,880) 



Variable 


Variable 


n 


% 


Highest Degree 


Other 


13 


0.48 


(Missing =150) 


Baccalaureate 


138 


5.05 




Masters 


2,527 


92.56 




Doctorate 


52 


1.90 




Total 


2,730 


100.00 


Geographic Region 


Cook/Lake 


595 


21.76 


(Missing = 146) 


Collar 


528 


19.31 




Northern 


243 


8.89 




West/Central 


154 


5.63 




East/Central 


173 


6.33 




South/West 


122 


4.46 




Southern 


132 


4.83 




Chicago 


787 


28.79 




Total 


2,734 


100.00 


Locale 


Large City & Fringe 


1,771 


64.68 


(Missing = 142) 


Mid City & Fringe 


412 


15.05 




Large & Small Town 


206 


7.52 




Rural 


349 


12.75 




Total 


2,738 


100.00 


Employment Status 


0-25% time employed 


10 


0.37 


(Missing =150) 


26-50% time employed 


49 


1.79 




51-75% time employed 


94 


3.44 




76-100% time employed 


2,577 


94.40 a 




Total 


2,730 


100.00 



a Full-time (100%) employment = 93.30% (n = 2,730) 



The Cochran-Mantel-Haenszel (CMH) statistic for general association was used 
to identify relationships between rows, columns, or strata. This statistic, used for nominal 



data, assumes an equal odds ratio and tests the null hypothesis that two variables are 



independent, given a third strata variable. In the current study, for example, this statistic 
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would test whether the conditional odds ratio of survey mode and response wave among 
females and males equals one. A statistically significant finding ,p< 0.05, would indicate 
that survey mode and response wave are not independent when controlling for gender. 
Such a finding would be followed with chi-square analyses controlling for each level of 
the categorical variable (Landis, Heyman, & Koch, 1978). A non-significant CMH for 
general association,/; > 0.05, would be followed with separate 2x2 chi-squares, one for 
the categorical variable by survey mode pooled over response wave, and one for the 
categorical variable by response wave pooled over survey mode. 

Assumptions . Regarding chi-square assumptions, observations were independent by the 
nature of the design by which data were collected. Subjects responded with either a mail 
or a web survey and either during Wave 1 or Wave 2, thus there was only one observation 
per subject and only one subject in each of the four respondent groups. Also, categories 
were mutually exclusive and observations were measured as frequencies. Subjects, 
however, were not randomly selected from the population, thus the external validity 
(generalizability) of interpretations from these chi-square analyses will be limited by this 
violation. 

Analysis of Variance (ANOVA) 

Since the design in the current study was nonorthogonal, descriptive statistics 
tables include weighted means and standard deviations for all conditions as well as 
adjusted means and standard errors for main effects (e.g., survey mode: Mail, Web; 
response wave: Wavel, Wave2). Continuous variables were analyzed with between- 
subjects, fixed effects, nonorthogonal (unequal ns) analyses of variance (ANOVA) F- 
tests. ANOVA is an inferential parametric statistic that tests the variance of means among 
two or more groups by comparing the ratio (F-ratio) of observed differences to an error 
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term (Ary et al., 2002). An important criterion of the ANOVA is that the continuous 
variable being measured must be at least at an interval level (i.e., not nominal or ordinal) 
(Harwell, 1988). To produce valid results, ANOVAs must also satisfy three assumptions: 
a) observations or scores must be independent within and between groups, b) groups must 
have equal (homogeneous) variances, and 3) group means must be normally distributed 
(Glass, Peckham, & Sanders, 1972; Penfield, 1994; Scheffe, 1959; Stevens, 1996). A test 
is considered robust if its actual Type I error rate (a), the probability of erroneously 
rejecting a true null hypothesis, is close to nominal significance level and its actual Type 
II error rate (statistical power), the probability of correctly rejecting a false null 
hypothesis, is close to theoretical power (Lix, Keselman, & Keselman, 1996). Violating 
ANOVA assumptions can affect Type I and Type II error rates (Glass et al., 1972; Lix et 
al., 1996). Glass et al. (1972) note, “The relevant question is not whether ANOVA 
assumptions are met exactly, but rather whether the plausible violations of the 
assumptions have serious consequences on the validity of probability statements based on 
the standard assumptions” (p. 237). 

Assumption of Independence . When using ANOVA, the assumption of independence is 
the assumption that scores or observations are not influenced by previous scores or other 
observations. This assumption is a function of a study‘s design and is addressed by using 
random sampling and random assignment techniques (Lix et al., 1996). Violating this 
assumption can negatively affect the Type I error rate and statistical power of the F-test 
(Glass et al., 1972; Scheffe, 1959; Stevens, 1996; Walsh, 1947). The assumption of 
independence in the current study was addressed by the nature of the design by which 
data were collected. Subjects responded with either a mail or a web survey and either 
during Wave 1 or Wave 2, thus there was only one observation per subject and only one 
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subject in each group (e.g., no repeated measures). The external validity of interpretations 
from ANOVAs may be limited, however, because subjects were not randomly selected 
from the population and were not randomly assigned to groups. Rather, all subjects in the 
sampling frame were invited to participate and subjects were pennitted to self-select the 
survey mode. 

Assumption of Homogeneity of Variance . Homogeneity of variance occurs when variance 
on a dependent variable is equal (homogeneous) across all levels of the independent 
variable. Heterogeneous variances in nonorthogonal ANOVAs can seriously affect a and 
reduce theoretical power; therefore, the ANOVA should not be used in this circumstance 
(Glass et ah, 1972). Glass et al. (1972) note that in nonorthogonal designs, the actual a 
becomes greater than nominal a when a smaller sample is paired with a larger variance, 
and actual a becomes less than nominal a when a smaller sample is paired with a smaller 
variance. Thus, a Levene’s Homogeneity of Variance test (Levene, 1960), which is also 
not dependent on the assumption of normality (Snedecor & Cochran, 1980), was 
conducted prior to each ANOVA to examine the homogeneity of group variances. Since 
SAS version 9.1 can conduct only a Levene’s test in a simple one-way ANOVA model 
(SAS Institute, 2004), a one (continuous response quality variable) by four (respondent 
group: Maill, Mail2, Webl, Web2) ANOVA was used with the “hovtest=levene” option 
in a proc glm means statement. A non-significant Levene’s test {p > 0.05) revealing equal 
variances was followed with a standard two-way ANOVA (survey mode by response 
wave) using Type-III sums of squares to test main effects and interactions. A significant 
Levene’s test (p < 0.05) revealing unequal variances was followed with a Welch’s Robust 
Test of Equality of Means ANOVA (Welch, 1947; Welch, 1951) with planned contrasts 
to test the variance in group means (in SAS, the Welch’s ANOVA is only available in the 
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one-way model). Standard and Welch’s ANOVAs revealing statistically significant 
interactions were followed with analyses of the simple effects. If a Welch’s ANOVA was 
used for the interaction analysis, then a Welch’s ANOVA was also used for the simple 
effects analysis given that the distribution had already been shown by the Levene’s test to 
be heterogeneous. 

Assumption of Normality . Normality is a measure of the distribution of scores relative to 
a normal (symmetrical) curve. In a normal curve, sample means and variances are 
statistically independent, meaning that the means and variances of repeated samples from 
the same normally distributed population would correlate to zero (Glass & Hopkins, 
1996). Nonnormal distributions are described in terms of skewness and kurtosis. 
Skewness refers to a distribution’s symmetry and indicates that one of its tails is longer 
than the other. A symmetrical distribution has a skewness of zero. A positively skewed 
distribution indicates the presence of a longer tail on the positive side of a curve (skewed 
to the right), which pulls the mean closer toward the extreme scores located on the right 
side of the distribution. A negatively skewed distribution has the opposite effect, pulling 
the mean toward extreme scores located on the negative (left) side of the distribution. 
Kurtosis refers to how peaked or flat a distribution is and how thick or thin its tails are 
relative to a nonnal (mesokurtic) distribution (kurtosis = 0). Leptokurtic distributions 
(kurtosis > 0) are highly peaked distributions with heavy or thick tails, while platykurtic 
distributions are relatively flat with light or thin tails (Ary et ah, 2002). Outliers were 
included in all data analyses, regardless of the statistics used, because there was no reason 
to believe that the data were invalid. 

ANOVAs are robust to nonnonnality (Box, 1953; Glass & Hopkins, 1996; Glass 
et ah, 1972; Hays, 1994; Kesehnan et al., 1998), even in nonorthogonal designs with 
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skewed distributions (Glass & Hopkins, 1996; Glass et al., 1972; Schneider & Penfield, 
1997); however, kurtosis can affect statistical power when sample sizes are small (Glass 
et al., 1972). In other words, even if normality assumptions are violated, inferences from 
ANOVAs do not appear to be seriously invalidated. On the other hand, the Welch’s 
ANOVA is less robust to nonnonnality (see “Analysis Using Welch’s ANOVA” below). 
Analysis Using Welch’s ANOVA . The Welch’s ANOVA, a parametric robust 
modification of the standard ANOVA, uses adjusted degrees of freedom and weighted 
variances instead of pooled variances (Algina & Olejnik, 1984). It is an acceptable 
alternative to the standard ANOVA when populations are normally distributed and 
variances are unequal (Algina, Ashima, & Lin, 1994; Clinch & Keselman, 1982; Dijkstra 
& Werter, 1981; Harwell, Rubinstein, Hayes, & Olds, 1992: Lix et al., 1996; Schneider & 
Penfield, 1997; Welch, 1947; Welch, 1951; Wilcox, Charlin, & Thompson, 1986; 

Zijlstra, 2004), especially when designs are nonorthogonal (Alyounes, 1999; Brown & 
Forsythe, 1974; Glass & Hopkins, 1996; Levy, 1978; Lix et al., 1996). While it is 
relatively robust to nonnormality in orthogonal designs (Bonett & Price, 2002; Fan, 

2006), it may be influenced by the shape of a distribution in nonorthogonal designs 
(Bonett & Price, 2002; Gans, 1981; Harwell et al., 1992). Error rates can be inflated in 
nonorthogonal designs, if data are highly skewed (skewness > 2) (Lix et al., 1996; 

Zijlstra, 2004) or if any of the group sizes is less than 10 (Lix et al., 1996). Welch’s 
appears to be robust to nonnonnality when the ratio of the largest to the smallest standard 
deviation is three or less (Brown & Forsythe, 1974; Wilcox et al., 1986). 

In circumstances where nonnonnal data may negatively affect Welch’s ANOVA 
results (e.g., skewness > 2, any of the group sizes is less than 10 or ratio of largest to 
smallest standard deviation < 4), data transformations are suggested to nonnalize a 
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nonnormal distribution (Lix et al., 1996). Nonnonnal distributions many times result 
from the presence of outliers. Some researchers suggest using Winsorized or trimmed 
means instead of the standard least squares statistics to reduce the effects of nonnonnality 
(Lix & Kesehnan, 1998; Wilcox, 1995). Such methods, however, that change or delete 
outliers create ethical concerns (Fraenkel & Wallen, 2006) and can change the null 
hypothesis from testing differences in group means to testing differences in trimmed 
means (Lix & Kesehnan, 1998). Data transfonnations to a different measurement scale 
(e.g., log or square root) are also suggested where necessary to achieve a more normal 
distribution (Snedecor & Cochran, 1980; Tabachnick & Fidell, 2001); however, the issue 
that such transfonnations can change the null hypothesis from testing differences in group 
means to testing differences in transformed means is still a concern (Lix & Kesehnan, 
1998). Where data transfonnations were ineffective, the significance level was shifted 
from a = 0.05 to a more stringent criterion, a = 0.025 to conect for distortions resulting 
from asymmetrical distributions (Keppel, 1991, pp. 97-98). See Alyounes (1999) for 
details regarding the methodology and behavior of the standard and Welch’s ANOVAs. 
/-test Analysis 

Two continuous response quality variables, anecdotal comments and multiple 
response use, occurred only in the mail survey, thus were analyzed only by response 
wave. These variables were analyzed with two-tailed /-tests to determine if there were 
differences in the means by response wave. To produce valid results, /-tests are subject to 
the same three assumptions as ANOVAs: a) observations or scores must be independent, 
b) groups must have equal (homogeneous) variances, and 3) group means must be 
normally distributed (Ary et al., 2002; Glass & Hopkins, 1996). As discussed above 
regarding the assumption of independence, subjects were not randomly selected from the 
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population, thus the external validity of interpretations from /-tests will be limited by this 
violation. The /-test is robust to nonnormality (Boneau, 1960; Glass et ah, 1972; 
Sawilowsky & Blair, 1992). According to Glass and Hopkins (1996), “...the condition of 
normality can be largely disregarded as a prerequisite for using the /-test. The /-test is 
robust with respect to failure to meet the nonnality assumption.” (p. 291). Thus, 
normality was not examined for these variables. Homogeneity of variance was examined 
prior to using the /-test. If equal variances were revealed,/? > 0.05, then the pooled 
method was used. If unequal variances were revealed, a < 0.05, then the Satterthwaite 
method was used. 

Logistic Regression Analysis 

Logistic regression is a specific type of regression analysis that uses the 
mathematical criterion of maximum likelihood rather than least squares as in multiple 
regression (Glass & Hopkins, 1996). Its outcome variable is dichotomous (binary) (Hair, 
Anderson, Tatham, & Black, 1998) and its predictor variables are continuous, discrete or 
both (Tabachnick & Fidell, 2001). Logistic regression uses a transformation (logit) to 
force the prediction equation to predict the odds of a discrete outcome (e.g., group 
membership) from a set of continuous, discrete, dichotomous, or mixed predictor 
variables. The predictors do not need to be normally distributed (Mertler & Vannatta, 
2005; Tabachnick & Fidell, 2001), have equal within-group variance, or be linearly 
related (Tabachnick & Fidell, 2001). While one of the variables in a logistic regression is 
an outcome, Tabachnick and Fidell (2001) caution that predictions on the outcome 
variable are not to be considered causal inferences. 

Logistic regression was used to in the current study to examine whether 
demographic and response quality variables performed differently in their ability to 
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predict survey mode or response wave. It was selected over discriminant function analysis 
(DFA), a comparable procedure, for the following reasons: 

1 . Logistic regression is more robust to assumption violations than discriminant 
function analysis (Press & Wilson, 1978; Tabachnick & Fidell, 2001). A DFA 
assumes that predictor variables are normally distributed, linearly related, and 
have equal covariance matrices (circumstances frequently violated in the 
current study); whereas a logistic regression is robust to these assumptions 
(Hair et ah, 1998; Press & Wilson, 1978; Tabachnick & Fidell, 2001). 
Additionally, the DFA is more sensitive to the presence of outliers, requiring 
either data transfonnations or that the outliers be eliminated (Tabachnick & 
Fidell, 2001, p.462). 

2. Logistic regression can handle categorical predictor variables much easier than 
DFA (Hair et ah, 1998). Logistic regression can include any mixture of 
nominal, ordinal, interval or ratio predictor variables (Tabachnick & Fidell, 
2001); however, DFA may not accurately predict the outcome variable when 
the explanatory variables are dichotomous (Press & Wilson, 1978). Rather, 
DFA sometimes overestimates the association for dichotomous predictors 
(Hosmer & Lemeshow, 2000). Given that the discriminant function in a DFA 
is a linear equation (e.g., a linear combination of the predictor variables), the 
observed variables should contain a linear relationship; however, this 
assumption is usually violated when the outcome variable is qualitative (Press 
& Wilson, 1978). 

3. Logistic regression correctly classifies a higher proportion of cases than DFA 
(Kester, Linton, & Sullivan, 2002; Press & Wilson, 1978). In a study 



97 




identifying at-risk students who would be successful in an alternative high 
school program, Kester et al. (2002) observed that logistic regression 
produced a predictive true positive hit rate that was 15% greater than DFA. 

4. A logistic regression’s regression coefficients permit estimating the odds 
ratios for each of the predictor variables (Cody & Smith, 1997). 

A decision needed to be made regarding whether to use a direct (standard), 
sequential, or stepwise (statistical) logistic regression model. A direct logistic regression, 
wherein all predictors enter the model simultaneously, is used when there are no specific 
hypotheses or expectations regarding the importance or order of predictor variables. This 
procedure permits examination of each predictor’s contribution as if each predictor 
entered the equation last (Tabachnick & Fidell, 2001). Interpretation difficulties can 
occur, however, when predictors are correlated (Tabachnick & Fidell, 2001). In the 
current study, an examination of correlations among eight continuous predictor variables 
revealed three variable pairs correlated at p < 0.05 and eight pairs correlated at p < 0.01 
(see Table 26). The sequential procedure forces predictors to be added into the model in a 
sequential order specified by the researcher (SAS Institute, 2004; Tabachnick & Fidell, 
2001); however, the researcher must know the order in advance of running the models. In 
the current study, the order that predictors should be entered into the model is not known 
in advance. The stepwise procedure automatically adds or removes predictors from the 
model using statistical criteria (likelihood ratio test) (Tabachnick & Fidell, 2001). Only 
variables adding a significant amount of prediction power are added or retained in the 
model, while variables that are redundant or highly correlated with another variable 
already in the model are eliminated (Mertler & Vannatta, 2005). Since it runs the risk of 
overfitting the model to noise in the data, it is suggested for use with more exploratory 
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analyses such as screening and hypothesis-generating (Tabachnick & Fidell, 2001). 
Therefore, the decision was made to use the stepwise procedure (selection=stepwise 
option in SAS proc logistic). 

Another important decision concerned the appropriate a level of significance. 
While the default in SAS is a = 0.05, Shtatland, Cain, and Barton (2001) suggest that 
level is inadequate for interpretation and prediction purposes. Hosmer and Lemeshow 
(2000) suggest using a less stringent criterion such as a = 0.15 or 0.20. Using Monte 
Carlo simulations, Lee and Koval (1997) found that the best a range is 0.05 to 0.40. For 
this analysis, a significance level of 0. 15 (slentry = 0.15 in SAS proc logistic) was 
selected for a variable to be entered into the model and a significance level of 0.40 (slstay 
= 0.40) was selected for a variable to stay in the model. 

Analysis of Research Questions 

Research Question One 

Research question one (RQ1) was: “Does a difference exist in demographic 
characteristics by survey mode or response wave?” The demographic variables were 
gender, ethnicity, age, and professional employment. The null hypothesis was that there is 
no difference in demographics by survey mode or response wave. The research sub- 
questions were: 

a) Does a difference exist in gender by survey mode or response wave? 

b) Does a difference exist in ethnicity by survey mode or response wave? 

c) Does a difference exist in age by survey mode or response wave? 

d) Does a difference exist in professional employment by survey mode or 
response wave? 
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Gender data were obtained from survey question 41 (“What is your gender?”) and 
ethnicity data were obtained from survey question 42 (What is your racial or ethnic 
background?”). Age data were obtained from survey question 43 (“What is your age?”) 
while professional employment data were obtained from the Teacher Service Record 
Report. Ethnicity, originally a seven-category variable (Black or African American, 
American Indian or Alaskan Native, Asian, White or Caucasian, Latino/Hispanic, Native 
Hawaiian or Other Pacific Islander, and Biracial or Multiracial), was collapsed into an 
artificial dichotomous variable (White, Nonwhite) to eliminate cells with expected 
frequencies of less than five observations in the chi-square analyses. Gender and ethnicity 
were analyzed using chi-squares. Age and professional employment were analyzed using 
ANOVAs. 

Research Question Two 

Research question two (RQ2) was: “Does a difference exist in response quality by 
survey mode or response wave?” The response quality variables examined by both survey 
mode and response wave were: pronoun use, item nonresponse, response extremity, yea- 
saying, item completion errors, response length, and response equivalency. The response 
quality variables examined by just response wave were: pronoun use in anecdotal 
comments, response length in anecdotal comments, and multiple response use. The null 
hypothesis was that that there is no difference in response quality by survey mode or 
response wave. The research sub-questions were: 

a) Does a difference exist in pronoun use by survey mode or response wave? 

b) Does a difference exist in pronoun use in anecdotal comments by response 
wave? 
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c) Does a difference exist in item nonresponse by survey mode or response 
wave? 

d) Does a difference exist in response extremity by survey mode or response 
wave? 

e) Does a difference exist in yea-saying by survey mode or response wave? 

f) Does a difference exist in item completion errors by survey mode or response 
wave? 

g) Does a difference exist in multiple response use by response wave? 

h) Does a difference exist in response length by survey mode or response wave? 

i) Does a difference exist in the response length of anecdotal comments by 
response wave? 

j) Does a difference exist in response equivalency by survey mode or response 
wave? 

RQ2a: Does a difference exist in pronoun use by survey mode or response wave? The 
personal pronouns examined include the first person singular “I” and the first person 
plural “we” including their contractions (e.g., I’d, I’ve, I’ll, I’m, we’d, we’ve, we’ll, 
we’d). Three open-ended survey items were available for analysis by survey mode and 
response wave: questions 45, 46, and 48 (see Figure 8). The total number of first person 
singular and first person plural pronouns in responses to each of these open-ended items 
was counted for each respondent, and then collapsed into one variable producing a total 
number of pronouns for each respondent. Analysis was conducted using ANOVAs. 
RQ2b: Does a difference exist in pronoun use in anecdotal comments by response wave? 
The web survey’s design prevented respondents from providing anecdotal comments; 
however, anecdotal comments were provided in some mail surveys. Pronoun use in 
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anecdotal comments was analyzed in the mail survey by response wave using a two-tailed 
/-test. The personal pronouns examined were the same as those examined for RQla: first 
person singular “I” and the first person plural “we” including their contractions (e.g., I’d, 
I’ve, I’ll, I’m, we’d, we’ve, we’ll, we’d). The total number of pronouns in anecdotal 
comments was counted for each mail respondent (n = 566), and then collapsed into one 
variable producing a total number of pronouns for each respondent. 

Figure 8. Survey Questions 45, 46, and 48 Used to Analyze Pronoun Use 

45. Please list the 2 most rewarding aspects of your job: 

1 . 



2 . 



46. Please list the 2 most significant challenges confronting school counselors: 
1 . 



2 . 



48. Additional Comments: 



RQ2c: Does a difference exist in item nonresponse by survey mode or response wave? 
Twenty-eight survey items were available to analyze item nonresponse by survey mode 
and response wave: questions 2-5, 8-10, 12-14, and 29-48. Counting each of their sub- 
items, a total of 130 sub-items were included. The analysis was set up such that if a 
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respondent answered “yes” to question two, “Do you work as a school counselor this year 
with students in any grade 6 through 12?” then that respondent should have also provided 
a response to each of 130 sub-items that followed question two. The number of instances 
was counted in which each respondent skipped one of the 130 sub-items that s/he was 
supposed to answer. The 130 sub-items were collapsed into one variable producing a total 
number of skipped items for each respondent. A greater score indicated more skipped 
items and thus greater item nonresponse. 

Analysis was conducted using a Welch’s ANOVA. Since a nonorthogonal 
Welch’s ANOVA may have inflated error rates if data are highly skewed (skewness > 2) 
(Lix et ah, 1996; Zijlstra, 2004), if any of the group sizes is less than 10 (Lix et ah, 1996), 
or if the ratio of the largest to the smallest standard deviation is four or more (Brown & 
Forsythe, 1974; Wilcox et ah, 1986), additional analyses was conducted to check these 
assumptions. 

RQ2d: Does a difference exist in response extremity by survey mode or response wave? 
Six survey items containing percent scaled responses (e.g., 1-25%, 26-50%, etc.) were 
selected for this analysis: questions 17, 24-25, 27-29. Counting each of their sub-items, a 
total of 44 sub-items with percent scaled responses were available for analysis. See Figure 
9 for an example of these item types. 

Item responses were recoded with a value of one assigned to a respondent’s use of 
the 76-100% category and a value of zero assigned for use of any of the remaining 
categories (as suggested by Naemi, 2006). Next, the number of extreme responses was 
counted for each respondent for each of the 44 sub-items. The sub-items were then 
collapsed into one variable producing a total number of extreme responses by survey 
mode and response wave. Analyses were conducted using ANOVAs. 
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Figure 9. Sample Item with Percent Scaled Response 



17. In general, what percentage of the students on your caseload participate in career 
assessment(s) during their secondary education? 



None 1-25% 26-50% 51-75% 76-100% NA 



Students with disabilities 


□ 


□ 


□ 


□ 


□ 


□ 


Students without disabilities 


□ 


□ 


□ 


□ 


□ 


□ 



RQ2e: Does a difference exist in yea-saying by survey mode or response wave? Eight 
survey items containing “Yes” and “No” responses were selected for this analysis: 
questions 33-40. Counting each of their sub-items, a total of 38 sub-items with yes-no 
responses were available for analysis. See Figure 10 for an example of two sub-items 
contained within one item. 

Figure 10. Sample Item with Yes and No Responses 



For questions 33-40, please indicate (1) whether you do the activity and (2) whether you 
consider the activity to be within the scope of your job. 



33. Review career planning materials and 


Do the activity 


Within scope of job 


practices for: 


Yes 


No 


Yes 


No 


Gender bias and/or “representation” 


□ 


□ 


□ 


□ 


Racial or ethnic bias and/or “representation” 


□ 


□ 


□ 


□ 



Item responses were recoded with a value of one assigned to use of the “yes” 
response and a value of zero assigned to use of the “no” response. Next, the number of 
yes responses was counted for each respondent for each of the 38 sub-items. The sub- 
items were then collapsed into one variable producing a total number of yes responses for 
each respondent. Analyses were conducted using ANOVAs. 

RQ2f) Does a difference exist in item completion errors by survey mode or response 
wave? Two separate analyses were conducted that examined item completion errors 
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related to two contingency-type items (questions 10 and 14) by survey mode and response 
wave. No automated skip, branching or error-checking features were used in the web 
survey for these items, thereby permitting a survey mode and response wave comparison. 

Question 10 analysis . The first analysis examined question 10, a contingency-type 
item, by survey mode and response wave. Respondents (n = 52) entering a zero for 
question 10 were indicating that that had no special education students on their caseload. 
Instructions and a definition in section two of the survey were, “Questions 17 through 28 
focus on students with and without disabilities on your caseload. If your caseload does 
not include specific groups of students, please indicate NA. Students with disabilities are 
students who receive special education services.” Therefore, if a respondent entered a 
zero in question 10, then the 13 subsequent items in the survey referring to students with 
disabilities on a respondent’s caseload should have been answered with an NA (questions 
12, 17-22, 24, 26-27, and 29-31). Counting each of their sub-items, a total of 70 sub- 
items were available for this analysis by survey mode and response wave. Figure 1 1 
presents question 10 followed by two examples of subsequent items for which a 
respondent should have selected NA. 

Items responses were recoded such that if a respondent entered a zero for question 
10 and then correctly entered an NA for a subsequent sub-item, the response was coded as 
a zero, otherwise, the response was coded as a one, indicating an item completion error. 
Next, the number of item completion errors was counted for each respondent for each of 
the 70 sub-items. The sub-items were then collapsed into one variable producing a total 
number of item completion errors for each respondent. 

Question 14 analysis . The second analysis examined question 14, a contingency- 
type item, by survey mode and response wave. Respondents (n =112) selecting “No” for 
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this item were instructed to “Skip to Section 3 - page 7” (question 29). These 
respondents, therefore, should not have provided responses for any of the 14 questions 
(questions 15-28) prior to question 29. Counting the sub-items for each of the 14 
questions, a total of 88 sub-items were available for this analysis by survey mode and 
response wave. Figure 12 presents question 14 as it appears on the survey. 

Figure 11. Survey Questions 10, 12 and 18 

10. Of the students on your total caseload this school year, how many receive special 
education services? 

ENTER NUMBER of special ed students on your caseload 

12. Please check how often you will meet individually with the students on your 
caseload during the school year: 

0-2 times 3-4 times Over 4 times NA 

Typical student with a disability □ □ □ □ 

Typical student without a disability □ □ □ □ 

18. How often does the typical student on your caseload participate in career 
assessment during his/her secondary education? 



None 


1-25% 


26-50% 


51-75% 


76-100% 


NA 


Students with disabilities □ 


□ 


□ 


□ 


□ 


□ 


Students without disabilities □ 


□ 


□ 


□ 


□ 


□ 



Figure 12. Sample Contingency Item, Question 14 

14. Generally, do students on your caseload participate in any career assessment(s) at 
some time during their secondary education ? (Check one) 

□ Yes □ No (Skip to Section 3 - page 7) 
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Item responses were recoded such that if a respondent entered a response to one of 
the 88 sub-items that should not have been answered, the response was coded as a one so 
it could be counted as an item completion error. Otherwise, if a respondent skipped the 
item, the response was coded as a dot to indicate a missing response. Next, the number of 
item completion errors was counted for each respondent for each of the 88 sub-items. The 
sub-items were then collapsed into one variable producing a total number of item 
completion errors for each respondent. Question 10 and question 14 analyses were both 
conducted using ANOVAs. 

RQ2g: Does a difference exist in multiple response use by response wave? Automated 
features programmed into the web survey prevented web respondents from providing 
multiple responses where only one response was expected; however, mail respondents 
provided multiple responses where such responses were not expected. In order to 
combine data from both survey modes, rules had to be applied in cases where mail 
respondents provided multiple responses so that only one response was retained for 
analysis. For example, in items with categorical responses in which respondents selected 
multiple categories, the response in the highest, most positive, or most personal category 
was retained (see Figure 13 for examples of these item types). In open-ended items with 
instructions stating how many responses to list (e.g., “list three,” “list up to five,”), only 
the first responses provided up to the requested number of responses were retained for 
analysis (e.g., if five responses were requested, only the first five were retained) and 
additional responses were not recorded. 

Analysis of the impact of applying rules to eliminate multiple responses is beyond 
the scope of this study; however, a two-tailed /-test analysis was conducted to determine 
if multiple response use in Maill differed statistically from Mail2. A variable was created 
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to count the number of additional responses that could not be recorded for each mail 
respondent (n = 566), producing a total number of multiple responses for each mail 
respondent. 

RQ2h: Does a difference exist in response length by survey mode or response wave? 
Three open-ended items were available for analysis by survey mode and response wave: 
questions 45-46, and 48. A substantial amount of data cleaning was required to convert 
typographical errors, symbols and variations of words and phrases into common forms so 
they could be counted equally among respondents (see Figure 14). 

Prior to data cleaning “self-esteem,” “selfesteem,” and “self esteem” would have 
been counted as one, one and two words respectively; “NCLB” and “No Child Left 
Behind” would have been counted as one and four words respectively; and “2x2,” “two- 
by-two,” and two by two” would have been counted as one, one and three words 
respectively. The Merriam-Webster Online Dictionary (Merriam-Webster, 2005) was 
consulted for proper word forms where possible. Figure 15 presents the variations of 
words that were assigned specific word length values. Punctuations markings (e.g., 
quotations, dashes, slashes, periods, etc.) were not included in response length counts. 

The number of words in each respondent’s response to each of the three items was 
counted. Then, the total number of words in each of the three items was collapsed into 
one variable producing a total response length for each respondent. 

Analysis was conducted using a Welch’s ANOVA. Since a nonorthogonal 
Welch’s ANOVA may have inflated error rates if data are highly skewed (skewness > 2) 
(Lix et ah, 1996; Zijlstra, 2004), if any of the group sizes is less than 10 (Lix et ah, 1996), 
or if the ratio of the largest to the smallest standard deviation is four or more (Brown & 
Forsythe, 1974; Wilcox et ah, 1986), an analysis was conducted to check these 
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assumptions. Since skewness exceeded the recommendations for a robust Welch’s 
ANOVA, a second analysis was conducted to determine if using a data transformation to 
normalize the skewed distribution produced different results (Lix et ah, 1996). A square 
root transformation was used to draw in the extreme scores on the positive side of the 
distribution (Snedecor & Cochran, 1980; Tabachnick & Fidell, 2001). 

RQ2i: Does a difference exist in the response length of anecdotal comments by response 
wave? Response length in anecdotal comments was analyzed in the mail survey by Wave 
1 and Wave 2 using a two-tailed /-test. The total number of words in each mail 
respondent’s response to each of three items (questions 45, 46, and 48) was counted. 

Then, the total number of words in each of the three items was collapsed into one variable 
producing a total response length for each respondent. 

RQ2j : Does a difference exist in response equivalency by survey mode or response wave? 
One survey item (question 29) containing 20 sub-items with percent scaled responses 
(None, 1-25%, 26-50%, 51-75%, 76-100%, NA) was available for analysis. The mean 
score of each of the 20 sub-items was calculated and then compared. See Figure 15 for an 
example of these item types. 

The response scale comprised the first category labeled “None,” followed by four 
categories with equal intervals each representing one quarter of 100%, followed by a sixth 
and final category labeled “NA.” Since a valid comparison of the means when using an 
ANOVA requires that the dependent variable being measured must be at least at an 
interval level (i.e., not nominal or ordinal) (Harwell, 1988), the two categories in the 
response scale that were not equal intervals with the remaining four were assigned a value 
of “missing” (e.g., a dot in SAS) and thus removed from the analysis. The remaining 
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categories retained their original assigned values as follows: 1-25% = 2, 26-50% = 3,51- 
75% = 4, and 76-100% = 5. 



Figure 13. Examples of Partial Items for Which Mail Respondents Provided Multiple Responses 
and the Rules Used to Determine the Retained Response 



Survey Item with Multiple Responses Selected 


Retained 

Response 


12. Please check how often you will meet individually with the students on 
your caseload during the school year: 

0-2 times 3-4 times Over 4 times NA 




Typical student with a disability 0 □ 0 □ 

18. How often does the typical student on your caseload participate in career 
assessment during his/her secondary education? 


0 Over 4 
times 
(greatest) 


None 1-25% 26-50% 5 1-75% 76-100% NA 




Students with disabilities □□00 □ □ 


051-75% 

(greatest) 


2 1 . In vour school, who has primary responsibility for implementing career 
assessment(s) for the students on your caseload? 




No one 1 do Other Person NA 




Students with disabilities □ 0 0 □ 


01 do (most 
personal) 


33. Review career planning materials and practices for: 




Yes No 

Gender bias and/or “representation” 0 0 


0 Yes (most 
positive) 



Of 880 total respondents, 729 respondents provided a response to one or more of 
the 20 sub-items in question 29, while 151 respondents skipped all of the sub-items in 
question 29. Of the 729 respondents who provided a response to one or more sub-items, 
29 were removed from the analysis because they provided no responses to any of the four 
interval-level categories (1-25%, 26-50%, 51-75%, 76-100%). Thus, 700 respondents 
were included in this analysis. 
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Figure 14. Variations of Words in Open-Ended Responses that Were Assigned Specific Word 
Length Values 



Variations of Words that were All Counted as One Word 


CRUISING COM 


OVER LOAD 


CASE LOAD 


PAPER WORK 


CITY WIDE 


PART TIME 


COPS, C.O.P.S. 


POST GRAD 


CO WORKER 


POST SECONDARY 


E G, E.G. 


PREK 


EMAIL 


PRE REGISTRATION 


EVERY DAY 


PRE SCHOOL 


FACE TO FACE 


PRE TEEN 


IE, I.E. 


RE ENTERING 


KEY ORG 


SELF ESTEEM 


MULTI CULTURAL 


STATE WIDE 


NON CERTIFIED 


WELL BEING 


NON TRADITIONAL 


WORK LOAD 


ONE ON ONE, ONE TO ONE, 1 TO 1 
OVER EMPHASIS 


WORK ROOM 


Variations of Words that were All Counted as Two Words 


ALOT 


LASTNAME 


CAREERKEY, CAREER-KEY 


PHONECALL 


CAREERCRUISING 


SCHOOL WIDE, SCHOOL-WIDE 


FIELDTRIP 


2X, 2-TIMES 


FOLLOWUP, FOLLOW-UP 


WORKKEY, WORK-KEY 



A variable was created to calculate a mean score for each of the 700 respondents 



included in the analysis. This score was obtained by adding the score for each sub-item 



into a total score, and then dividing by the total number of sub-items that each respondent 



answered. A lower mean score indicated that the respondent’s answers were closer to the 



lower end of the scale (1-25% category = 2) and a higher mean score indicated that 



responses were closer to the higher end of the scale (76-100% category = 5). A mean 



score was used rather than a total score for each respondent to pennit comparing the 



collapsed 20 sub-items to the original scale. For example, the mean score range was two 



to five, the same as the original scale, whereas the total score range was two to 100. 
Analyses were conducted using ANOVAs. 
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Figure 15. Sample Item with Percent Scaled Response Options 



29. For each of the following, please indicate the proportion of students on your caseload with 
whom you typically implement or include in the activity. 



Students with disabilities 




Students without disabilities 




None 1- 26- 51- 76- NA 


None 


1- 


26- 


51- 


76- 


NA 


25% 50% 75% 100% 




25% 


50% 


75% 


100% 




Administer career assessment(s) 














□ 

□ 

□ 

□ 

□ 

□ 


□ 


□ 


□ 


□ 


□ 


□ 


Provide information regarding results of 
career assessments 














□ 

□ 

□ 

□ 

□ 

□ 


□ 


□ 


□ 


□ 


□ 


□ 


Assist students to identify career goals 














□ 

□ 

□ 

□ 

□ 

□ 


□ 


□ 


□ 


□ 


□ 


□ 


Provide information about careers and/or 
options 














□ 

□ 

□ 

□ 

□ 

□ 


□ 


□ 


□ 


□ 


□ 


□ 



Research Question Three 

Research question three (RQ3) was: “Do demographic characteristics and 
response quality predict survey mode or response wave?” The null hypothesis was that 
demographic characteristics and response quality don’t predict survey more or response 
wave. The research sub-questions were: 

a. Do demographic characteristics and response quality predict survey mode? 

b. Do demographic characteristics and response quality predict response wave? 
Binary stepwise logistic regressions (selection = stepwise in SAS proc logistic) 

with a requirement of 0.15 to enter the model and 0.40 to stay in the model (slentry = 0.15 
and slstay = 0.40) were used to examine whether demographic and response quality 
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variables performed differently in predicting survey mode or response wave. The 
categorical predictor variables were gender (coded as male = 0 and female =1) and 
ethnicity (coded as Nonwhite = 0 and White = 1). The continuous predictor variables 
were age, professional employment, pronoun use, response extremity, yea-saying, item 
completion errors, response length, and response equivalency. Given that the item 
completion error variable (RQ2f) had such a small number of cases (n = 52 for Question 
10 and n = 1 12 for Question 14) compared to other variables in the model having ns of 
700 or greater, the item completion error variable was not included in these analyses. The 
outcome variables were survey mode (coded as mail = 0 and web = 1) and response wave 
(coded as Wave 1 = 1 and Wave 2 = 2). In the logistic regression predicting survey mode, 
response wave was added to the predictors, and in the logistic regression model 
predicting response wave, survey mode was added to the predictors. All 1 1 predictors 
were entered into each model. 
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CHAPTER IV 



RESULTS 

Chapter 4 presents the results of data analysis for this study and is arranged into 
six sections. The first section presents response rates for the overall sample as well as by 
survey mode and response wave. The second and third sections address sample 
representativeness and respondent demographics respectively. The fourth section presents 
the results of analyses for research question one and its four sub-questions. The fifth 
section presents the results for research question two and its 10 sub-questions. The last 
section presents the results for research question three and its two sub-questions. 

Response Rates 

School counselors in Illinois public schools counseling students in grades 6-12 
comprised the target population of 3,019 potential participants identified via an Illinois 
State Board of Education Teacher Service Record report. After accounting for 
participants unable to participate (e.g., extended illness, maternity leave, retirement, no 
longer employed, incorrect address), the adjusted accessible population was 2,880 Illinois 
public school counselors counseling students in grades 6-12 during the 2004-05 academic 
year. 

Total survey response rate was 35.69% (n = 1,028). After accounting for 
respondents who returned blank, incomplete, unusable, and multiple surveys, the adjusted 
response rate was 30.56% (n = 880). Table 5 presents the distribution of response rates by 
survey mode and response wave. Maill respondents had the highest response rate (n = 
433, 49.20%) and Web2 (n = 63, 7.16%) had the lowest. Differences between response 
conditions were not statistically significant, % (1, N= 880) = 1.38,/? = 0.24. Table 6 
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presents actual and expected frequencies for response rates by survey mode and response 



wave. 



Table 5 



Descriptive Statistics for Response Rate by Survey Mode and Response Wave (N= 880) 



n % 



Survey Mode 



Mail 


566 


64.32 


Web 


314 


35.68 


Response Wave 
Wave 1 


684 


77.73 


Wave 2 


196 


22.27 


Survey Mode x Response Wave 
Maill 


433 


49.20 


Mail2 


133 


15.11 


Webl 


251 


28.52 


Web2 


63 


7.16 


Total 


880 


100.00 



Table 6 



Actual and Expected Frequencies for Response Rates by Survey Mode and Response Wave (N = 
880) 



Response 

Conditions 


Actual 

n % 


Expected 

(2x2 y 2 , Mode by Wave) 
n % 


Expected 

(Goodness-of-Fit) 

n % 


Maill 


433 


49.20 


439.94 


49.99 


220 


25.00 


Mail2 


133 


15.11 


126.06 


14.33 


220 


25.00 


Webl 


251 


28.52 


244.06 


27.73 


220 


25.00 


Web2 


63 


7.16 


69.94 


7.95 


220 


25.00 


Total 


880 




69.94 


100.00 


880 


100.00 




100.00 










Result 




X 2 (1,7V= 880) = 


1.38, p = 0.24 


r (3, N= 880) 


= 357.04,p <0.0001 



Sample Representativeness 



Table 7 presents the results of chi-square (%") goodness-of-fit tests, which were 



used to examine the representativeness of the sample as compared to the population. Four 



categorical variables were available for this analysis: highest degree, geographic region, 



locale, and professional employment. Sample statistics for survey respondents were 



compared to population parameters from the Teacher Service Record report provided by 
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the Illinois State Board of Education. For this analysis, the population parameter size (n = 
2,738) is lower than the accessible population size (n = 2,880) and the sample statistic 
size (n = 808) is lower than the actual sample size (n = 880) due to missing data (n = 

142). 

Some of the levels in the “highest degree” variable were collapsed to eliminate 
cells with expected frequencies of less than five observations in the chi-square analysis. 
The “Up To Bachelor” category is comprised of two original categories: “None” and 
“Baccalaureate.” The “Master & Other Advanced” category is comprised of two original 
categories: “Certificate of Advanced Study, Specialist, and Six- Year Certificate” and 
“Masters.” Some levels of the locale variable were also collapsed to create cells with a 
minimum of five observations. For example, “Farge City” and “Urban Fringe of Farge 
City” were collapsed into “Farge City & Fringe,” “Mid-size City” and “Urban Fringe of 
Mid-size City” were collapsed into “Mid City & Fringe,” and “Farge Town” and “Small 
Town” were collapsed into “Farge & Small Town.” The continuous variable, professional 
employment, was separated into nine live-year increments for analysis. 

Chi-square goodness of fit analyses revealed that the sample did not differ 
statistically from the population by highest degree, % (2, N = 805) = 1.60, p = 0.45, or by 
professional employment, % (8, N = 804) = 10.55 , p = 0.23. Thus, the null hypothesis 
that there is no difference in highest degree between the sample and the population, and 
the null hypothesis that there is no difference in professional employment between the 
sample and the population were both not rejected at a < 0.05. 
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Table 7 



Chi-square Goodness-of-Fit Analysis of Sample Representativeness 



Variable 


Sample 
n % 


Population 
n % 


x 2 


df 


P 


Highest Degree 
















Up to Bachelor 


34 


4.22 


141 


5.16 


1.60 


2 


0.45 


Master & Other Advanced 


754 


93.66 


2537 


92.93 








Doctorate 


17 


2.11 


52 


1.90 








Total 


805 


100.00 


2730 


100.00 








Missing 


3 




8 










Geographic Region 
















Cook/Lake 


170 


21.07 


595 


21.76 


42.58 


7 


<0.0001 


Collar 


172 


21.31 


528 


19.31 








Northern 


97 


12.02 


243 


8.89 








West/Central 


65 


8.05 


154 


5.63 








East/Central 


54 


6.69 


173 


6.33 








South/West 


35 


4.34 


122 


4.46 








Southern 


50 


6.20 


132 


4.83 








Chicago 


164 


20.32 


787 


28.79 








Total 


807 


100.00 


2734 


100.00 








Missing 


1 




4 










Locale 
















Large City & fringe 


468 


57.92 


1771 


64.68 


24.34 


3 


<0.0001 


Mid City & fringe 


121 


14.98 


412 


15.05 








Large & Small Town 


82 


10.15 


206 


7.52 








Rural 


137 


16.96 


349 


12.75 








Total 


808 


100.00 


2734 


100.00 








Missing 


0 




0 










Professional Employment 
















0-5 years 


94 


11.69 


299 


10.96 


10.55 


8 


0.23 


6- 1 0 years 


113 


14.05 


378 


13.86 








11-15 years 


141 


17.54 


471 


17.27 








1 6-20 years 


112 


13.93 


354 


12.98 








21-25 years 


91 


11.32 


268 


9.83 








26-30 years 


128 


15.92 


429 


15.73 








31-35 years 


95 


11.82 


389 


14.26 








36-40 years 


28 


3.48 


117 


4.29 








4 1 & up years 


2 


0.25 


22 


0.81 








Total 


804 


100.00 


2723 


100.00 








Missing 


4 




11 











2 

The sample does not represent the population by geographic region, (7, N = 



807) = 42.58,/? < 0.0001, or by locale. 



X 2 0, 



N = 808) = 24.34,/? < 0.0001. Thus, the null 



hypothesis that there is no difference in geographic region between the sample and the 



population, and the null hypothesis that there is no difference in locale between the 
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sample and the population were both rejected at a < 0.05. Chicago is the most under- 
represented geographic region (sample = 20.32%, population = 28.79%, difference = 
8.47%) and Northern is the most over-represented geographic region (sample = 12.02%, 
population = 8.89%, difference = 3.13%). Given that Chicago was the most under- 
represented geographic region, it follows that Large City & Fringe was the most 
underrepresented locale (sample = 57.92%, population = 64.68%, difference = 6.76%). 
Rural was the most over-represented locale (sample = 16.96%, population = 12.75%, 
difference = 4.21%). 

In summary, the study sample appears to represent the population based on 
highest degree and professional employment but not based on geographic region and 
locale. 

Respondent Demographics 

Descriptive statistics for respondent demographics by survey mode and response 
wave are presented in Table 8 for gender and ethnicity. Respondents were 75.03% female 
(, n = 553) and 24.97% male (n = 184) with more females than males represented in each 
of the four respondent groups as well as for the overall sample. Regarding ethnicity, the 
majority of respondents were White (86.40%, n = 629) with more Whites than any other 
ethnic group represented in each of the four respondent groups and for the overall sample. 
Black/African Americans comprised the second largest ethnic group (9.48%, n = 69) and 
all of the other ethnic groups combined comprised the remaining 4. 12% (n = 30) of the 
ethnic distribution. Inferential statistics for gender and ethnicity are presented in the 
results for research question one. 
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Table 8 



Descriptive Statistics for Gender and Ethnicity by Survey Mode and Response Wave (N = 880) 





Wave 1 


Wave 2 




Total 


Gender 


n 


% 


n 


% 


n 


% 


Female 














Mail 


311 


42.20 


107 


14.52 


418 


56.72 


Web 


104 


14.11 


31 


4.21 


135 


18.32 


Subtotal 


415 


56.31 


138 


18.72 


553 


75.03 


Male 














Mail 


106 


14.38 


22 


2.99 


128 


17.37 


Web 


44 


5.97 


12 


1.63 


56 


7.60 


Subtotal 


150 


20.35 


34 


4.61 


184 


24.97 


Total 














Mail 


417 


56.58 


129 


17.50 


546 


74.08 


Web 


148 


20.08 


43 


5.83 


191 


25.92 


Total 


565 


76.66 


172 


23.34 


737 


100.00 


Ethnicity 


White/Caucasian 














Mail 


359 


49.31 


103 


14.15 


462 


63.46 


Web 


129 


17.72 


38 


5.22 


167 


22.94 


Subtotal 


488 


67.03 


141 


19.37 


629 


86.40 


Black/African American 














Mail 


41 


5.63 


19 


2.61 


60 


8.24 


Web 


8 


1.10 


1 


0.14 


9 


1.24 


Subtotal 


49 


6.73 


20 


2.75 


69 


9.48 


Latino/Hispanic 














Mail 


8 


1.10 


4 


0.55 


12 


1.65 


Web 


6 


0.82 


1 


0.14 


7 


0.96 


Subtotal 


14 


1.92 


5 


0.69 


19 


2.61 


Asian 














Mail 


4 


0.55 


2 


0.27 


6 


0.82 


Web 


1 


0.14 


1 


0.14 


2 


0.27 


Subtotal 


5 


0.69 


3 


0.41 


8 


1.10 


Native Elawaiian/Pacific Islander 














Mail 


0 


0.00 


0 


0.00 


0 


0.00 


Web 


2 


0.27 


0 


0.00 


2 


0.27 


Subtotal 


2 


0.27 


0 


0.00 


2 


0.27 


Biracial/Multiracial 














Mail 


0 


0.00 


1 


0.14 


1 


0.14 


Web 


0 


0.00 


0 


0.00 


0 


0.00 


Subtotal 


0 


0.00 


1 


0.14 


1 


0.14 


Native American/ Alaskan Native 














Mail 


0 


0.00 


0 


0.00 


0 


0.00 


Web 


0 


0.00 


0 


0.00 


0 


0.00 


Subtotal 


0 


0.00 


0 


0.00 


0 


0.00 


Total 














Mail 


412 


56.59 


129 


17.72 


541 


74.31 


Web 


146 


20.05 


41 


5.63 


187 


25.69 


Total 


558 


76.65 


170 


23.35 


728 


100.00 



Note : missing =143 for Gender and missing =152 for Ethnicity. 
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Table 9 presents descriptive statistics for age and professional employment 
demographics by survey mode and response wave. Respondents had an age range of 25- 
73 years with a mean age of 47.82 years ( SD = 10.32). Maill respondents were the oldest 
(M= 49.36, SD = 9.94) and Web2 were the youngest (M= 42.03, SD = 1 1.52). 

Respondents had a professional employment range of 1-43 years with a mean of 
18.58 years (SD = 10.08). Maill respondents had the most years of professional 
employment (M= 19.64, SD = 9.99) and Webl had the least (M= 16.69, SD = 9.95). 
Inferential statistics for age and professional employment are presented in the results for 
research question one. 

Table 9 



Descriptive Statistics for Age and Professional Employment by Survey Mode and Response Wave 
(N = 880) 







Age (n = 707) 




Professional Employment (n z 


= 806) 




n 


M 


SD 


M A dj 


SE Adj 


n 


M 


SD 


M A di 


SE Ad j 


Survey Mode 
Mail 


524 


49.19 


9.90 


49.00 


0.52 


530 


19.54 


9.94 


19.42 


0.52 


Web 

Resp. Wave 


183 


43.89 


10.52 


43.21 


0.91 


276 


16.73 


10.12 


16.82 


0.79 


Wave 1 


546 


48.05 


10.24 


46.88 


0.49 


641 


18.59 


10.07 


18.16 


0.41 


Wave 2 
Mode x Wave 


161 


47.03 


10.60 


45.33 


0.93 


165 


18.55 


10.15 


18.08 


0.86 


Maill 


402 


49.36 


9.94 






413 


19.64 


9.99 






Mail2 


122 


48.63 


9.81 






117 


19.20 


9.78 






Webl 


144 


44.39 


10.22 






228 


16.69 


9.95 






Web2 


39 


42.03 


11.52 






48 


16.96 


10.96 






Total 


707 


47.82 


10.32 






806 


18.58 


10.08 







Note : Age missing =173 and Professional Employment missing = 74. 



In summary, the majority of respondents in each of the four respondent groups as 



well as for the overall sample were White (n = 629, 86.40%) and female (n = 553, 
75.03%). Respondents had a mean age of 47.82 years (SD = 10.32) with a mean of 18.58 



years (SD = 10.08) of professional employment. 



120 




Results of Research Questions 



Research Question One 

Research question one (RQ1) was: “Does a difference exist in demographic 
characteristics by survey mode or response wave?” The research sub-questions were: 

a) Does a difference exist in gender by survey mode or response wave? 

b) Does a difference exist in ethnicity by survey mode or response wave? 

c) Does a difference exist in age by survey mode or response wave? 

d) Does a difference exist in professional employment by survey mode or 
response wave? 

RQla: Does a difference exist in gender by survey mode or response wave? See Table 8 
for descriptive statistics for gender. Analysis of gender by survey mode and response 
wave was conducted using a 2x2x2 chi-square. The CMH statistic for general association 
revealed no statistically significant relationship between rows, columns or strata, % ( 1 , N 
= 737) = 0.04,/? = 0.84. Separate 2x2 analyses followed, revealing no statistically 
significant differences for gender by survey mode pooled over response wave, x ( 1 . A = 
737) = 2.61,/? = 0.1 1, or for gender by response wave pooled over survey mode, % (1 ,N 
= 737) = 3.24,/? = 0.07. Thus, the null hypothesis that there is no difference in gender by 
survey mode or response wave was not rejected at a < 0.05. See Table 10 for chi-square 
actual and expected frequencies for gender. 

RQlb: Does a difference exist in ethnicity by survey mode or response wave? See Table 8 
for descriptive statistics for ethnicity. Analysis of ethnicity by survey mode and response 
wave was conducted using a 2x2x2 chi-square. Ethnicity, originally a seven-category 
variable was collapsed into a dichotomous variable (White, Nonwhite) for analysis. The 
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CMH statistic for general association revealed no statistically significant relationship 
between rows, columns or strata, % (1, N= 728) = 0.21 ,p = 0.64. Separate 2x2 analyses 
followed, revealing no statistically significant differences for ethnicity by survey mode 
pooled over response wave, ( 1 , N = 728) = 1 .8 1 , p = 0. 1 8, or for ethnicity by response 
wave pooled over survey mode, % (1 ,N = 728) = 2.26 , p = 0.13. Thus, the null 
hypothesis that there is no difference in ethnicity by survey mode or response wave was 
not rejected at a < 0.05. See Table 1 1 for chi-square actual and expected frequencies for 
ethnicity. 



Table 10 

Actual and Expected Frequencies for Gender by Survey Mode and Response Wave (N = 880) 







n 


Wave 1 
Expected 


n 


Wave 2 
Expected 


n 


Total 

Expected 


Female 


Mail 


311 


313.69 


107 


104.31 


418 


409.69 




Web 


104 


101.31 


31 


33.69 


135 


143.31 




Total 


415 


423.94 


138 


129.06 


553 




Male 


Mail 


106 


104.35 


22 


23.65 


128 


136.31 




Web 


44 


45.65 


12 


10.35 


56 


47.69 




Total 


150 


141.06 


34 


42.94 


184 




Total 


Mail 


417 




129 




546 






Web 


148 




43 




191 






Total 


565 




172 




737 





Note : missing =143. 



RQlc: Does a difference exist in age by survey mode or response wave? See Table 9 for 



descriptive statistics for age. Levene’s test revealed homogeneity of group variances, F( 3, 



703) = 0.93 ,p = 0.42, so a standard ANOVA followed. The ANOVA indicated a 



statistically significant difference by survey mode, F (\ , 703) = 30.59 ,p< 0.0001, but not 



by response wave, F( 1, 703) = 2.19,/? = 0.14, and no interaction, F{ 1, 703) = 0.61,/ = 



0.44. Thus, the null hypothesis that there is no difference in age by survey mode or 
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response wave was rejected at a < 0.05. Mail respondents ( M Adj = 49.00, SE Adj = 0.52) 
were older than web respondents (M Adj = 43.21, SE Adj = 0.91). 

Table 1 1 

Actual and Expected Frequencies for Ethnicity by Survey Mode and Response Wave (N = 880) 







n 


Wave 1 
Expected 


n 


Wave 2 
Expected 


n 


Total 

Expected 


White 


Mail 


359 


358.44 


103 


103.56 


462 


467.43 




Web 


129 


129.56 


38 


37.44 


167 


161.57 




Total 


488 


482.12 


141 


146.88 


629 




Nonwhite 


Mail 


53 


55.86 


26 


23.14 


79 


73.57 




Web 


17 


14.14 


3 


5.86 


20 


25.43 




Total 


70 


75.88 


29 


23.12 


99 




Total 


Mail 


412 




129 




541 






Web 


146 




41 




187 






Total 


558 




170 




728 





Note : missing =152. 



RQld: Does a difference exist in professional employment by survey mode or response 
wave? See Table 9 for descriptive statistics for professional employment. Levene’s test 
revealed homogeneity of group variances, F( 3, 802) = 0.67, p = 0.57, so a standard 
ANOVA followed. The ANOVA indicated a statistically significant difference by survey 
mode, F( 1, 802) = 7.44, p = 0.01, but not by response wave, F{ 1, 802) = 0.01 ,p = 0.93, 
and no interaction, F( \ , 802)= 0.14,/; = 0.71. Thus, the null hypothesis that there is no 
difference in professional employment by survey mode or response wave was rejected at 
a < 0.05. Mail respondents (M Adj = 19.42, SE Adj = 0.52) had more years of professional 
employment than web respondents ( M Adj = 16.82, SE tdj = 0.79). 

RQ1 Summary . Results revealed that gender and ethnicity did not differ by survey mode 
or response wave; however, both age and professional employment differed by survey 
mode but not by response wave (see Table 12). Mail respondents (M Adj = 49.00, SE AdJ = 
0.52) were older than web respondents (M Adj = 43.21, SE Adj = 0.91) and mail respondents 
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(M Adj = 19.42, SE Adj = 0.52) had more years of professional employment than web 
respondents (M Adj = 16.82, SE Adj = 0.79). 

Research Question Two 

Research question two (RQ2) was: “Does a difference exist in response quality by 
survey mode or response wave?” The research sub-questions were: 

a) Does a difference exist in pronoun use by survey mode or response wave? 

b) Does a difference exist in pronoun use in anecdotal comments by response 
wave? 

c) Does a difference exist in item nonresponse by survey mode or response 
wave? 

d) Does a difference exist in response extremity by survey mode or response 
wave? 

e) Does a difference exist in yea-saying by survey mode or response wave? 

f) Does a difference exist in item completion errors by survey mode or response 
wave? 

g) Does a difference exist in multiple response use by response wave? 

h) Does a difference exist in response length by survey mode or response wave? 

i) Does a difference exist in the response length of anecdotal comments by 
response wave? 

j) Does a difference exist in response equivalency by survey mode or response 
wave? 
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Summary of Results for Research Question One 
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RQ2a: Does a difference exist in pronoun use by survey mode or response wave? Table 



13 presents descriptive statistics for pronoun use by survey mode and response wave. 



Respondents had a pronoun use range of 0-7 pronouns with an overall mean of 0.3 1 (SD 



= 0.86) pronouns. Web2 respondents provided the most pronouns (M= 0.44, SD = 1.12) 



and Maill provided the least (M = 0.25, SD = 0.71). 

Table 13 

Descriptive Statistics for Pronoun Use by Survey Mode and Response Wave (N = 880) 





n 


M 


SD 


M Ad j 


SE A dj 


Survey Mode 
Mail 


566 


0.28 


0.75 


0.31 


0.04 


Web 


314 


0.37 


1.02 


0.40 


0.06 


Response Wave 
Wave 1 


684 


0.29 


0.82 


0.30 


0.03 


Wave 2 
Mode x Wave 


196 


0.40 


0.96 


0.41 


0.07 


Maill 


433 


0.25 


0.71 






Mail2 


133 


0.38 


0.88 






Webl 


251 


0.35 


0.99 






Web2 


63 


0.44 


1.12 






Total 


880 


0.31 


0.86 







Levene’s test indicated homogeneity of group variances, F(3, 876) = 1.69 ,p = 



0.17, so a standard ANOVA followed. The ANOVA revealed no statistically significant 



main effects by survey mode, F( 1, 876) = 1.23 ,p = 0.27, or by response wave, F(l, 876) 



= 2.21, p = 0.13, and no interaction, F( 1, 876) = 0.03 ,p = 0.86. The null hypothesis that 



that there is no difference in pronoun use by survey mode or response wave was not 



rejected at a < 0.05. 



RQ2b: Does a difference exist in pronoun use in anecdotal comments by response wave? 



See Table 14 for descriptive statistics for pronoun use in anecdotal comments by response 



wave. Respondents had a pronoun use range of 0-5 pronouns with a mean of 0. 15 (SD = 



0.56) pronouns. 
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Table 14 



Descriptive Statistics for Pronoun Use in Anecdotal Comments by Response Wave (N = 566) 





n 


M 


SD 


Maill 


433 


0.15 


0.57 


Mail2 


133 


0.16 


0.53 


Total 


566 


0.15 


0.56 



Results using a /-test for equal variances (pooled method) revealed no statistically 
significant difference in total pronoun use by response wave, t(564) = -0.14,/? = 0.89. 
Thus, the null hypothesis that there is no difference in pronoun use in anecdotal 
comments by response wave was not rejected at a < 0.05. 

RQ2c: Does a difference exist in item nonresponse by survey mode or response wave? 
Table 15 presents descriptive statistics for item nonresponse by survey mode and 
response wave for all 130 sub-items. Of 880 total respondents, three respondents were 
removed from the analysis because they skipped question two, which was used to identify 
the remaining items each respondent was supposed to answer. The item nonresponse 
range was 0-129 sub-items with a mean of 28.07 ( SD = 42.49) sub-items skipped. Webl 
respondents skipped the greatest number of items (M= 53.49, SD = 55.64) and Maill 
skipped the least (M= 14.58, SD = 25.13). 

Levene’s test indicated heterogeneity of group variances, F(3, 873) = 1 17.19,/? < 
0.0001, so a Welch’s ANOVA was used in place of the standard ANOVA. Results using 
the Welch’s ANOVA revealed a statistically significant interaction, Fw e ich(3, 209.20) = 
41.33,/? < 0.0001, prompting analysis of the simple effects. Statistically significant 
differences were revealed among survey modes for Wave 1, Fw e ich(l, 310.40) = 109.75,/? 
< 0.0001, and for Wave 2, F We ich(l, 77.09) = 14.98,/? = 0.0002, but not among response 
waves for mail, AweichO, 209.70) = 0.49,/? = 0.48, or for web, AweichO, 99.70) = 1.76,/? = 
0.19. Thus, the null hypothesis that there is no difference in item nonresponse by survey 
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mode or response wave was rejected at a < 0.05. There were more items skipped among 



Webl (M= 53.49, SD = 55.64) versus Maill (M= 14.58, SD = 25.13) respondents, and 



more items skipped among Web2 (M= 43.54, SD = 52.64) versus Mail2 (M= 16.39, SD 



= 26.26) respondents. 

Table 15 

Descriptive Statistics for Item Nonresponse by Survey Mode and Response Wave (N = 880) 





n 


M 


SD 


M-Adi 


SE Adj 


Survey Mode 
Mail 


563 


15.01 


25.38 


15.49 


1.92 


Web 


314 


51.50 


55.11 


48.52 


2.73 


Response Wave 
Wave 1 


682 


28.90 


43.45 


34.04 


1.54 


Wave 2 
Mode x Wave 


195 


25.16 


38.90 


29.97 


2.96 


Maill 


431 


14.58 


25.13 






Mail2 


132 


16.39 


26.26 






Webl 


251 


53.49 


55.64 






Web2 


63 


43.54 


52.64 






Total 


877 


28.07 


42.49 







Note : missing = 3 (3 respondents were removed from the analysis because 
they skipped question two, which was used to identify the remaining skipped 
items). 



Since a nonorthogonal Welch’s ANOVA was used, an additional analysis was 



conducted to check assumptions. Results revealed that skewness = 1.50, the smallest 



group size = 63, and the ratio of the largest (Webl, SD = 55.64) to the smallest (Maill, 



SD = 25.13) standard deviation = 2.21, all within the recommendations for a robust 
Welch’s ANOVA. 

RQ2d: Does a difference exist in response extremity by survey mode or response wave? 



Table 16 presents descriptive statistics for response extremity by survey mode and 



response wave. Mean use of the extreme response option was 19.16 (SD = 1 1.49). Maill 



respondents selected the most extreme responses (M= 19.78, SD = 1 1.30) and Web2 



selected the least (M= 17.60, SD = 1 1.86). 
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Table 16 



Descriptive Statistics for Response Extremity by Survey Mode and Response Wave (N = 880) 





n 


M 


SD 


M Adi 


SE A dj 


Survey Mode 
Mail 


553 


19.58 


11.44 


19.36 


0.58 


Web 


196 


17.98 


11.57 


17.85 


0.99 


Response Wave 
Wave 1 


576 


19.33 


11.38 


18.94 


0.54 


Wave 2 
Mode x Wave 


173 


18.61 


11.88 


18.27 


1.01 


Maill 


423 


19.78 


11.30 






Mail2 


130 


18.94 


11.91 






Webl 


153 


18.09 


11.53 






Web2 


43 


17.60 


11.86 






Total 


749 


19.16 


11.49 







Note: missing =131. 



Levene’s test indicated homogeneity of group variances, F( 3, 745) = 0.40, p = 



0.76, so a standard two-way ANOVA followed. The ANOVA revealed no statistically 



significant main effect by survey mode, F (\ , 745) = 1.74 ,p = 0.19, or by response wave, 



F(l, 745) = 0.34, p = 0.56, and no interaction, F (\ , 745) = 0.02, p = 0.88. Thus, the null 



hypothesis that there is no difference in response extremity by survey mode or response 
wave was not rejected at a < 0.05. 

RQ2e: Does a difference exist in yea-saying by survey mode or response wave? Table 17 



presents descriptive statistics for yea-saying by survey mode and response wave. Mean 
use of the yes response option was 14.55 (SD = 9.66). Web2 respondents selected the 
most yes responses (M= 15.43, SD = 1 1.56) and Maill selected the least (M= 14.22, SD 
= 9.35). 

Levene’s test indicated homogeneity of group variances, F( 3, 702) = 2. 15,/; = 



0.09, so a standard two-way ANOVA followed. The ANOVA revealed no statistically 



significant main effect by survey mode, F(l, 702) = 1.25, p = 0.26, or by response wave, 



F (\ , 702) = 0.01, p = 0.91, and no interaction, F(l, 702) = 0.00, p = 0.98. Thus, the null 



129 




hypothesis that there is no difference in yea-saying by survey mode or response wave was 
not rejected at a < 0.05. 

Table 17 

Descriptive Statistics for Yea-Saying by Survey Mode and Response Wave (N = 880) 





n 


M 


SD 


M Adj 


SE Adj 


Survey Mode 
Mail 


517 


14.25 


9.49 


14.29 


0.50 


Web 


189 


15.37 


10.09 


15.39 


0.85 


Response Wave 
Wave 1 


543 


14.52 


9.44 


14.78 


0.47 


Wave 2 
Mode x Wave 


163 


14.63 


10.37 


14.89 


0.87 


Maill 


396 


14.22 


9.35 






Mail2 


121 


14.36 


9.96 






Webl 


147 


15.35 


9.67 






Web2 


42 


15.43 


11.56 






Total 


706 


14.55 


9.66 







Note : missing = 174. 



RQ2f: Does a difference exist in item completion errors by survey mode or response 



wave? Table 18 presents descriptive statistics for item completion errors by survey mode 



and response wave for both contingency questions 10 and 14. 



For question 10 analysis, of 880 total respondents, 52 respondents indicated that 



they had no special education students on their caseload, but then provided responses to 



one or more of 70 subsequent sub-items. Mean item completion errors among the 52 



respondents with no special education students on their caseload was 8.98 ( SD = 13.35). 



Maill respondents had the most item completion errors (M= 12.36, SD = 15.86) and 



Mail2 had the least (M= 2.25, SD = 3.54). 



Levene’s test indicated homogeneity of group variances, F(3, 48) = 1.95, p = 0.13, 



so a standard two-way ANOVA followed. The ANOVA revealed no statistically 



significant main effect by survey mode, F( \ , 48) = 0.07, p = 0.80, or by response wave, 



F (\ , 48) = 1 .33, = 0.25, and no interaction, F {\ , 48) = 0.78, = 0.38. 
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Descriptive Statistics for Item Completion Errors for Questions 10 and 14 by Survey Mode and Response Wave (N = 880) 



to 



CM 



K 

I 

|5fc 

0 I 
o 



p 

to 



w 

lO^ ' 



Os 


oo 




CN 












in 


vq 


y — < 


O 














’ 1 




CN 












cn 


r- 


in 


in 












sq 


in 


Os 


CN 














© 


cn 














in 


CN 


o 


so 


N 


Os 


OO 


© 


o 


© 


© 


00 


so 


• — i 


N 


os 


© 


r- 


<N 


so 


P 


in 


N" 


N 


so 


o 


o 


*" H 




T “ H 












r “ < 


os 




in 


© 


N 


O 


cn 


o 


N" 


r- 


00 


CN 


cn 


N 


in 


' — i 


© 


in 


in 


© 






SO 


CN 




© 


CN 






in 


N 


N 


N" 


oo 


CN 


CN 


so 


in 


00 


<N 


N" 




cn 














K 



N" 






IN 












so 


CN 


CN 


p 












CN 


N" 


CN 


N" 












o 




CN 


Os 












CN 


© 


in 


N 












N 


so 


Os 


CN 












IN 


so 


OO 


CN 


SO 


N" 


IN 


N 


IN 


so 


N 


p 


N 


00 


in 


p 


in 


CN 


N" 


Os 


N" 


N" 


in 


CN 


O 


N 


CN 


’ 1 




’ 1 




' 1 




’ 1 




’ 1 




N" 


so 


Os 


so 


IN 


Os 


CN 


OO 


1 


p 


in 


© 


CN 


CN 


so 


CN 


os 


© 


so 


© 


CN 


CN 


CN 


so 


IN 


oo 


T-H 




T-H 




T-H 












so 






OO 


OO 


CN 


CN 


CN 


CN 




N" 




CN 




in 





131 




For question 14 analysis, of 880 total respondents, 1 12 respondents selected “no” 
for question 14, indicating that students on their caseload do not participate in career 
assessments at some time during their secondary education. These respondents were 
instructed to skip to section 3, page 7; however, continued to answer one or more of 88 
subsequent sub-items in section two. Mean item completion errors among these 1 12 
respondents was 3.54 ( SD = 10.70). Maill respondents had the most item completion 
errors (M= 6.77, SD = 14.17) and Web2 had the least (M= 0.00, SD = 0.00). 

Levene’s test indicated homogeneity of group variances, F(3, 108) = 2.18 ,p = 
0.10, so a standard two-way ANOVA followed. The ANOVA revealed no statistically 
significant main effect by survey mode, F(\ , 108) = 3.08, p = 0.08, or by response wave, 
F{ 1, 108) = 1.36, p = 0.25, and no interaction, F{ 1, 108) = 0.46 , p = 0.50. Thus, the null 
hypothesis that there is no difference in item completion errors by survey mode or 
response wave was not rejected at a < 0.05 for both question 10 and question 14 analyses. 
RQ2g: Does a difference exist in multiple response use by response wave? See Table 19 
for descriptive statistics. Mean multiple response use among 564 mail respondents was 
2.22 additional responses (SD = 4.16). 

Table 19 

Descriptive Statistics for Multiple Response Use in Mail Surveys by Response Wave (N = 566) 





n 


M 


SD 


Maill 


432 


2.30 


4.22 


Mail2 


132 


1.98 


3.98 


Total 


564 


2.22 


4.16 



Note : missing = 2. 



Results for equal variances revealed no statistically significant difference in 



multiple response use by response wave, t(562) = 0.75,/? = 0.45. Thus, the null 
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hypothesis that there is no difference in multiple response use by response wave was not 
rejected at a < 0.05. 

RQ2h: Does a difference exist in response length by survey mode or response wave? 
Table 20 presents descriptive statistics for response length by survey mode and response 
wave for all three items. Mean response length was 34.70 words ( SD = 24.77). Webl 
respondents provided the greatest number of words in their responses (M= 45.81, SD = 
38.38) and Maill provided the least (M= 31.31, SD = 18.08). 



Table 20 

Descriptive Statistics for Response Length by Survey Mode and Response Wave (N = 880) 





n 


M 


SD 


M Adj 


SE Adj 


Survey Mode 
Mail 


523 


31.55 


18.71 


31.82 


1.25 


Web 


180 


43.86 


35.65 


41.62 


2.13 


Response Wave 
Wave 1 


539 


35.02 


25.66 


38.56 


1.19 


Wave 2 
Mode x Wave 


164 


33.63 


21.62 


34.88 


2.16 


Maill 


401 


31.31 


18.08 






Mail2 


122 


32.33 


20.72 






Webl 


138 


45.81 


38.38 






Web2 


42 


37.43 


23.92 






Total 


703 


34.70 


24.77 







Note : missing = 111. 



Levene’s test indicated heterogeneity of group variances, F( 3, 699) = 12.37, p < 
0.0001, so a Welch’s ANOVA was used in place of the standard ANOVA. Results using 
the Welch’s ANOVA revealed a statistically significant interaction, Fw e ich(3, 144.10) = 



6.65, p = 0.0003, prompting analysis of the simple effects. A statistically significant 



difference was revealed among survey modes for Wave 1, F We ich(l, 158.40) = 18.31 ,P< 



0.0001, but not for Wave 2, Fw e ich(l, 63.47) = 1.52, p = 0.22, and not among response 



waves for mail, F We ich(l, 180.60) = 0.24,/? = 0.63, or for web, TweichO, 101.10) = 2.89,/? 



= 0.09. Thus, the null hypothesis that there is no difference in response length by survey 
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mode or response wave was rejected at a < 0.05. Response length was greater among 
Webl (M= 45.81, SD = 38.38) compared to Maill (M= 31.31, SD = 18.08) respondents. 

Since a nonorthogonal Welch’s ANOVA was used, an additional analysis was 
conducted to check assumptions. Results revealed that skewness = 2.58, the smallest 
group size = 42, and the ratio of the largest (Webl, SD = 38.38) to the smallest (Maill, 
SD = 18.08) standard deviation = 2.12. Thus, the skewness exceeded the 
recommendations for a robust Welch’s ANOVA, prompting a secondary analysis using 
data transformed by square root. Results revealed that skewness = 0.97, the smallest 
group size = 42, and the ratio of the largest (Webl, SD = 2.48) to the smallest (Maill, SD 
= 1.57) standard deviation = 1.17, all within the recommendations for a robust Welch’s 



ANOVA. Descriptive statistics are presented in Table 21. 

Table 21 

Descriptive Statistics for the Square Root of Response Length by Survey Mode and Response 
Wave (N = 880) 





n 


M 


SD 


Ma di 


SE Ad j 


Survey Mode 
Mail 


523 


5.39 


1.59 


5.40 


0.09 


Web 


180 


6.19 


2.35 


6.07 


0.16 


Response Wave 
Wave 1 


539 


5.61 


1.89 


5.84 


0.09 


Wave 2 
Mode x Wave 


164 


5.54 


1.72 


5.64 


0.16 


Maill 


401 


5.37 


1.57 






Mail2 


122 


5.43 


1.68 






Webl 


138 


6.30 


2.48 






Web2 


42 


5.85 


1.82 






Total 


703 


5.59 


1.85 







Note : missing = 177. 



Levene’s test using the square root of the response length indicated heterogeneity 



of group variances, F(3, 699) = 12.70,/? < 0.0001, so a Welch’s ANOVA was used in 



place of the standard ANOVA. Results using the Welch’s ANOVA revealed a statistically 



significant interaction, F We ich(3, 148.20) = 6.18,/? = 0.001, prompting analysis of the 
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simple effects. A statistically significant difference was revealed among survey modes for 
Wave l,F We ich(l, 176.00)= 16.97,/? < 0.0001, but not for Wave 2, F We i ch (l, 66.49) = 

1 .66, p = 0.20, and not among response waves for mail, F We i C h(l, 189.80) = 0.13,/? = 

0.71, or for web, F We i ch (l, 91.63) = 1.66,/? = 0.20. The square root of the response length 
was greater among Webl (M= 6.30, SD = 2.48) compared to Maill (M= 5.37, SD = 

1.57) respondents. Although the second analysis used data transformed by square root and 
the first analysis didn’t, the results of the two analyses were consistent. Thus, the 
hypothesis that there is no difference in response length by survey mode or response wave 
was rejected at a < 0.05. 

RQ2i: Does a difference exist in the response length of anecdotal comments by response 
wave? See Table 22 for descriptive statistics for response length in anecdotal comments 
by survey mode and response wave. Respondents had a response length range of 0-64 
words with a mean response length of 4.12 words ( SD = 9.29). 

Table 22 

Descriptive Statistics for Response Length in Anecdotal Comments by Response Wave (N = 566) 





n 


M 


SD 


Maill 


433 


4.09 


9.64 


Mail2 


133 


4.23 


8.09 


Total 


566 


4.12 


9.29 



Results for unequal variances (Satterthwaite method) revealed no statistically 



significant difference in response length by response wave, t(257) = -0.16,/? = 0.87. Thus, 



the null hypothesis that there is no difference in response length in anecdotal comments 



by response wave was not rejected at a < 0.05. 



RQ2j : Does a difference exist in response equivalency by survey mode or response wave? 



Table 23 presents descriptive statistics for response extremity by survey mode and 



response wave. The mean response score among respondents who responded to at least 
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one of the 20 sub-items (n = 700) was 4.34 ( SD = 0.84). The mean score in all conditions 



is between the 51-75% and 76-100% categories. Maill respondents provided the highest 



mean score (M= 4.39, SD = 0.80) and Webl provided the lowest (M= 4.26, SD = 0.92). 



Table 23 

Descriptive Statistics for Response Equivalency by Survey Mode and Response Wave (N = 880) 





n 


M 


SD 


M Adj 


SE Adj 


Survey Mode 
Mail 


521 


4.37 


0.81 


4.35 


0.04 


Web 


179 


All 


0.92 


4.27 


0.07 


Response Wave 
Wave 1 


536 


4.36 


0.83 


4.33 


0.04 


Wave 2 
Mode x Wave 


164 


4.30 


0.86 


4.29 


0.08 


Maill 


399 


4.39 


0.80 






Mail2 


122 


4.31 


0.84 






Webl 


137 


4.26 


0.92 






Web2 


42 


4.28 


0.93 






Total 


700 


4.34 


0.84 







Note : missing = 180. 



Levene’s test indicated homogeneity of group variances, F(3, 696) = 1.04, p = 



0.37, so a standard two-way ANOVA followed. The ANOVA revealed no statistically 



significant main effect by survey mode, F (\ , 696) = 0.73, p = 0.39, or by response wave, 



F(l, 696) = 0.14, = 0.71, and no interaction, F (\ , 696) = 0.35, p = 0.56. Thus, the null 



hypothesis that there is no difference in response extremity by survey mode or response 



wave was not rejected at a < 0.05. 



RQ2 Summary. Results revealed that there were no differences in pronoun use, pronoun 



use in anecdotal comments, response extremity, yea-saying, item completion errors, 



multiple response use, response length in anecdotal comments, and response equivalency 



by survey mode or response wave and there were. Significant differences were revealed in 



item nonresponse by survey mode at Wave 1 and by survey mode at Wave 2, but not by 



response wave for mail surveys or by response wave for web surveys. Webl respondents 
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had greater item nonresponse (M= 53.49, SD = 55.64) than Mail 1 (M= 14.58, SD = 
25.13) and Web2 had greater item nonresponse ( M= 43.54, SD = 52.64) than Mail2 (M = 
16.39, SD = 26.26). A significant difference was also revealed for response length by 
survey mode at Wave 1, but no differences were revealed for survey mode at Wave 2, 
response wave for mail, or response length for web. Using data transfonned by square 
root, Webl respondents had greater response length (M= 6.30, SD = 2.48) than Maill (M 
= 5.37, SD = 1.57). See Table 24 for a summary of the results for research question two. 
RQ2 Supplemental Analysis. A supplemental analysis was conducted using /-tests on the 
variables in RQ2b (pronoun use in anecdotal comments), RQ2g (multiple response use), 
and RQ2i (response length in anecdotal comments) to detennine if instances of these 
variables were statistically different than zero (h0=0 option in SAS proc ttest). Results revealed 
that each of these was statistically different than zero: pronoun use in anecdotal comments, 
t(565) = 6.42 ,p < 0.0001; multiple response use, t(563) = 12.69 ,p< 0.0001; and 
response length in anecdotal comments, t(565) = 10.55,/? < 0.0001. 

Research Question Three 

Research question three (RQ3) was: “Do demographic characteristics and 
response quality predict survey mode or response wave?” The research sub-questions 
were: 

a) Do demographic characteristics and response quality predict survey mode? 

b) Do demographic characteristics and response quality predict response wave? 
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Summary of Results for Research Question Two 



P P 

on on 



'-O 

1414 

A A A A 

^ CP ^ CP 

d d 
> *T) > <N 
l/A l/A 



d d 

O d 

d d 

d d 



^ § 



r- m so 

(N h 00 

o o o 



o Os so 00 00 

II H IT) 00 (S o\ o\ 

" © © © © © o 



J2 ^ ^ ^ 

5 t-P cn' 
J5 (N (N O 

h ri o 



(5 a, a, a. a, a, a. 

• c\ c\ cs c\ cs cs 

d - d - cn >o o . 
■ i r-- m o cn o © 



L_H SO so so 

^ r- r- r- 



Q\ ^ IT) <N CN CN 
G\ d" d" d" O O O 
^ r- r- r- r- r- 



K ^ &J 



c 



o 

d 


3 




d 

> 


> 

oj 


Vi/ 

> 

03 


'3 

a 


-g 


& 

o o -s 


2 




d 

£ 


£ 


£ 


£ 


,u S S3 
fa ffl 

« *s £ 


X 




X 

d 


o 


W 


d 


(§) 

d 


S3 fl cs 

2 •§ £ 


'3 

s 


CD 

s 


nd 

o 

S 


-a 

o 

a 


-a 

o 

a 


> 

d 

£ 


> 

d 

£ 


a B x 


, f- 


^d 












w u u 


CZ) 


d 




< 


< 


< 


< 


"O > "O 

o ca o 

a £ a 


d 

v 


o 


^ d 


> 


> 


> 


> 


■Ka 

(D 


*-H 

CLh 

'd 


O o O 

z « z 


o 

z 


o 

z 


o 

z 


< < < 




d 


<3 a 


< 


< 


< 


< 


> > > 


'3 

-4— » 


o 




-d 


43 


43 


43 


o o o 


1 

o 


o 

o , 


A c 


d 


o 


o 


o 


z z z 


£ 


Mh 


0) 

3> 


13 


"5 

3> 


"5 

3> 


"5 

3> 



<<<<<<<< 

>>>>>>>> 

oooooooo 

zzzzzzzz 

<<<<<<<< 



O (- 1 o 

•Su ?,Z 



138 



(Question 10) ANOVA, mode x wave interaction F( 1, 48) = 0.78, p = 0.38 

ANOVA, mode main effect F(\, 1 08) = 3.08, /? = 0.08 

(Question 14) ANOVA, wave main effect F{\, 108) = 1.36 , p = 0.25 No differences 

ANOVA, mode x wave interaction A(l, 108) = 0,46 , p = 0.50 




Table 24 (continued) 
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ANOVA, mode x wave interaction jjjT, 696) = 0.35, p = 0.56 




Table 25 presents a summary of basic descriptive statistics for all of the variables 
included in the logistic regression models. For detailed statistics, descriptions, and 
behaviors of these variables by survey mode and response wave, see the descriptive 
statistics tables and analyses for research questions one and two earlier in this chapter as 
the variables selected as predictors in this analysis are the same variables as used to 



answer research questions one and two. 

Table 25 

Descriptive Statistics for Variables Used in Logistic Regression Models 



Variable 


Description 


n 


M 


SD 


Gender 


0 = Male, 1 = Female 


737 


0.75 


0.43 


Ethnicity 


0 = Nonwhite, 1 = White 


728 


0.86 


0.34 


Survey Mode 


0 = Mail, 1 = Web 


880 


0.36 


0.48 


Response Wave 


0 = Wave 1, 1 = Wave 2 


880 


1.22 


0.42 


Age 


Range 25 -73 years 


707 


47.82 


10.32 


Profess. Employment 


Range 1-43 years 


806 


18.58 


10.08 


Pronoun Use 


Range 0-7 pronouns 


880 


0.31 


0.86 


Item Nonresponse 


Range 0-129 items skipped 


877 


28.07 


42.49 


Response Extremity 


Range 0-44 extreme responses 


749 


19.16 


11.49 


Yea Saying 


Range 0-38 yes responses 


706 


14.55 


9.66 


Response Length 


Range (square root) 1.41 - 14.90 
words 


703 


5.59 


1.85 


Response Equivalency 


2 = 1-25%, 3 = 26-50%, 4 = 51-75%, 
5 = 76-100% 


700 


4.34 


0.84 



Table 26 presents Pearson Product Moment correlations between continuous 



predictor variables. An examination of correlations among eight continuous predictor 
variables revealed three variable pairs correlated at p < 0.05 and eight pairs correlated at p 
< 0 . 01 . 
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Pearson Correlations Between Continuous Predictor Variables (N = 880) 
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RQ3a: Do demographic characteristics and response quality predict survey mode? Of 880 
respondent observations, 565 were retained in the model (missing = 315). All 1 1 
predictors were entered into the model predicting survey mode, with mail coded as zero 
and web coded as one. The probability modeled was the mail survey mode, (using event = 
‘0’ on the left side of the model statement in SAS proc logistic). Prior to the first step in 
the stepwise logistic regression analysis, the intercept-only model was fitted, Wald % (1, 



N = 565) = 129.86,/i < 0.0001. See Table 27 for initial model estimates. 

Table 27 

Analysis of Effects Prior to Entry into the Logistic Regression Model Predicting Mail Survey 
Mode, All Predictors Included 



Effect 


df 


Score y 2 


P 


Response Wave (Wave 1 = 0, Wave 2 = 
1) 

Gender (Male = 0, Female = 1) 


1 


0.29 


0.59 


1 


3.03 


0.08 


Ethnicity (Nonwhite = 0, White =1) 


1 


2.00 


0.16 


Age 


1 


24.84 


<.0001 


Professional Employment 


1 


13.30 


0.0003 


Pronoun Use 


1 


9.75 


0.0018 


Item Nonresponse 


1 


1.63 


0.20 


Response Extremity 


1 


0.76 


0.38 


Yea-Saying 


1 


5.46 


0.02 


Response Length (square root) 


1 


13.35 


0.0003 


Response Equivalency 


1 


0.61 


0.43 



2 

In step one, the model with intercept and age was significant, Wald yf (1, N = 



565) = 23.83,/? < 0.0001, and age was not removed. In step two, response length was 
added. In the model with intercept, age and response length, both predictors were 
significant,/? < 0.0001, and neither predictor was removed. In step three, gender was 
added to the model: intercept, age ip < 0.0001), response length ip < 0.0001), and gender 
ip = 0.01), and all met the criterion of 0.40 to stay in the model. In step four, yea-saying 
was added to the model: intercept, age (p < 0.0001), response length (p < 0.0001), gender 
(p = 0.01), and yea-saying (p = 0.05) and none was removed. In step five, ethnicity was 
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added to the model: intercept, age (p < 0.0001), response length (p < 0.0001), gender (p = 
0.01), yea-saying (p = 0.03), and ethnicity (p = 0.09) and none was removed. In step six, 
response extremity was added to the model: intercept, age (p < 0.0001), response length 
( p < 0.0001), gender (p = 0.01), yea-saying (p = 0.01), ethnicity (p = 0.06), and response 
extremity (p = 0.13) and none was removed. None of the remaining variables met the 
criterion of 0. 15 for entry into the model, so the stepwise selection was terminated. 

Table 27 presents a summary of the final stepwise logistic regression model 
predicting the mail survey mode. Four of the six variables retained in the final model 
reached the level of significance for this study: age, response length, gender and yea- 
saying. Response wave, ethnicity, professional employment, pronoun use, item 
nonresponse, response extremity, and response equivalency were not significant 
predictors. Using parameter estimates from Table 28, the prediction equation for the log 
of returning a mail survey is: 0.28 + 0.05(Age) - 0.25(Response Length) + 0.58(Gender) - 
0.03(Yea-Saying). Inserting the values of each variable into this equation produces the 
log (odds) predicting a mail survey. 

The odds ratio (OR) for age indicates that when holding all other variables 
constant, the probability that a respondent will self-select a mail survey increases 1 .06 
times for every one unit increase in age. The OR for response length indicates that when 
holding all other variables constant, the probability that a respondent will self-select a 
mail survey decreases by 22% for every unit increase in response length, e.g., an OR of 

0. 78. In other words, an increasing response length is associated with decreasing odds of 
self-selecting a mail survey. The OR of 1.79 for gender, coded as male = 0 and female = 

1, indicates that when holding all other variables constant, females are more 1.79 times 
more likely than males to self-select a mail survey. The OR for yea-saying indicates that 
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Inal Logistic Regression Model Predicting Mail Survey Mode from Age, Response Length, Gender, Yea-Saying, Ethnicity, and Response 
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when holding all other variables constant, the probability that a respondent will self-select 
a mail survey decreases about 3% for every unit increase in yea-saying, e.g., an OR of 
0.97. In other words, as yea-saying increases, the odds of self-selecting a mail survey 
decrease. 

Model fit statistics were used to examine how well the model fit the data. A 
Likelihood Ratio test of the full model versus the null model was statistically significant, 
y 2 (6, N= 565) = 56.73, p< 0.0001, and accounted for approximately 14% of the 
variance (Nagelkerke R = 0. 14) associated with whether or not a respondent returned a 
mail or a web survey. Results of the Hosmer-Lemeshow goodness-of-fit test revealed no 
evidence of a lack of fit for the final model, yj (8, N= 565) = 5.46, p = 0.71. 

RQ3b: Do demographic characteristics and response quality predict response wave? Of 
880 respondent observations, 565 were retained in the model (missing = 315). All 1 1 
predictors were entered into the first model predicting response wave, with Wave 1 coded 
as zero and Wave 2 coded as one. The probability modeled was the Wave 1 response 
wave. Prior to the first step in the stepwise logistic regression analysis, the intercept-only 
model was fitted, Wald y 2 (l, N= 565) = 165.60 ,p< 0.0001. See Table 29 for initial 
model estimates. No predictors obtained the level required for entry into the model, thus 
the stepwise selection procedure terminated. 
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Table 29 



Analysis of Effects Prior to Entry into the Logistic Regression Model Predicting Wave 1, All 
Predictors Included 



Effect 


df 


Score y 2 


P 


Survey Mode (Mail = 0, Web = 1) 


1 


0.29 


0.59 


Gender (Male = 0, Female =1) 


1 


1.13 


0.29 


Ethnicity (Nonwhite = 0, White = 
1) 

Age 


1 


0.37 


0.54 


1 


0.18 


0.67 


Professional Employment 


1 


0.00 


1.00 


Pronoun Use 


1 


0.08 


0.78 


Item Nonresponse 


1 


1.45 


0.23 


Response Extremity 


1 


0.00 


0.97 


Yea-Saying 


1 


0.43 


0.51 


Response Length (square root) 


1 


0.39 


0.53 


Response Equivalency 


1 


0.15 


0.70 



RQ3 Summary. Results revealed that age and gender had significant positive partial 



effects while response length and yea-saying had significant negative partial effects in 
predicting mail survey mode. No variables predicted response wave. 
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CHAPTER V 



DISCUSSION 

Chapter 5 begins with a summary and discussion of the findings from this study. 
This section reviews the results presented in Chapter 4 in the context of the field of 
survey research as presented in the literature. Second, the study’s limitations are 
addressed. Finally, the study’s practical implications and recommendations for further 
research are presented. 

The purpose of this nonexperimental, quantitative study was to examine 
differences in demographics and response quality among mail and web surveys and two 
data collection cycles when respondents self-select the survey mode. This study was 
designed to contribute knowledge to the field regarding differences in demographics and 
response quality between mail and web surveys, web and mail mixed-mode designs using 
simultaneous mode delivery, differences between early and late responders, and mode 
self-selection. 

Results revealed that mail respondents were older and had more years of 
professional employment than web respondents, item nonresponse was greater in web 
than in mail surveys, and response length was greater in web versus mail surveys at Wave 
1 . Age, gender, response length, and yea-saying had significant partial effects in 
predicting the mail survey mode. No differences in survey mode or response wave were 
observed for gender, ethnicity, pronoun use, response extremity, yea-saying, item 
completion errors, and response equivalency. No differences by response wave were 
observed in pronoun use and response length for anecdotal comments and in multiple 
response use; however, each of these three variables was statistically different than zero. 
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Finally, no variables were observed to predict response wave. The next section 
summarizes key findings in the context of the literature. 

Response Rate 

Total survey response rate was 35.69% (n = 1,028) and the adjusted response rate 
was 30.56% ( n = 880). The survey mode response rate was 64.32% (n = 566) for the mail 
mode versus 35.68% (n = 3 14) for the web mode. The response wave response rate was 
77.73% (n = 684) for Wave 1 versus 22.27% (n = 196) for Wave 2. Results revealed no 
differences between actual and expected response rates in a mode by wave chi-square 
comparison, but a goodness-of-fit chi-square (using 25% and n = 220 for each cell) 
revealed over-representation in both survey modes at Wave 1 and under-representation in 
both modes at Wave 2. 

The survey mode response rate distribution was consistent with the literature 
(Kiesler & Sproull, 1986; Shih & Fan, 2007b; Underwood et ah, 2000), even among 
studies involving mode self-selection and mail mode initial contacts (Shih & Fan, 2007b). 
The finding of no statistically significant difference between survey modes was consistent 
with Jun’s (2005) survey about health behaviors among 1,000 undergraduate university 
students. Regarding response wave, although the two response waves comprised 
approximately the same length of time, the finding of over-representation in Wave 1 and 
under-representation in Wave 2 was expected given that Wave 1 included the initial 
contact and first three reminders, whereas Wave 2 comprised only the fourth reminder. It 
is typically accepted in the survey research field that most respondents will respond 
during the initial and first follow-up phases of data collection (Dillman, 2000). 
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Discussion of Research Questions 



Research Question One 

Research question one was, “Does a difference exist in demographic 
characteristics by survey mode or response wave?” The majority of respondents in each 
of the four respondent groups as well as for the overall sample were female (n = 553, 
75.03%) and White (n = 629, 86.40%), with a mean age of 47.82 years ( SD = 10.32) and 
a mean of 18.58 years (SD = 10.08) of professional employment. Results revealed no 
differences in gender and ethnicity by survey more or response wave; however, both age 
and professional employment differed by survey mode but not by response wave. Mail 
respondents were older and had more years of professional experience than web 
respondents. 

The gender distribution was consistent with the literature for mail and electronic 
surveys (Ayers, 2004; Borkan, 2006; McCabe et ah, 2002; Witte et ah, 2000) as well as 
for surveys involving school counselors in particular (Berry, 2006; Fitch & Marshall, 
2004; Fritz, 2004; Ruebensaal, 2006; Young, 2004). The nonsignificant findings for 
gender and ethnicity by survey mode were consistent with the literature (Borkan, 2006; 
Hayslett & Wildemuth, 2004; Kiesler & Sproull, 1986). The findings for age were 
consistent with four studies (Kaufman et al., 1997; Palmquist & Stueve, 1996; Schmidt, 
1997; Zhang, 2000) and professional employment findings were consistent with Kim et 
al. (2000) and Hollowell et al. (2000). Findings suggest that mail and web surveys 
provided similar coverage for gender and ethnicity. Given the survey mode differences in 
age and professional experience, however, the mail and web mixed-mode design may 
provide coverage to a wider age and professional experience range than using either of 
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the two modes separately. Carrying the data collection cycle into the new school year 
appeared to have no effect on demographics. 

Research Question Two 

Research question two was, “Does a difference exist in response quality by survey 
mode or response wave?” Results revealed that there were no differences in pronoun use, 
response extremity, yea-saying, item completion errors, and response equivalency by 
survey mode or response wave. There were also no differences in pronoun use in 
anecdotal comments, multiple response use, and response length in anecdotal comments 
in mail surveys by response wave. Supplemental analysis revealed that pronoun use in 
anecdotal comments, multiple response use, and response length in anecdotal comments 
was each statistically different than zero. 

Significant differences were revealed in item nonresponse by survey mode at 
Wave 1 and by survey mode at Wave 2, but not by response wave for mail surveys or by 
response wave for web surveys. Webl respondents had greater item nonresponse (M = 
53.49, SD = 55.64) than Mail 1 (M= 14.58, SD = 25.13) and Web2 had greater item 
nonresponse (M= 43.54, SD = 52.64) than Mail2 (M= 16.39, SD = 26.26). A significant 
difference was also revealed for response length by survey mode at Wave 1, but no 
differences were revealed for survey mode at Wave 2, response wave for mail, or 
response length for web. Using data transformed by square roots, Webl respondents had 
greater response length (M= 6.30, SD = 2.48) than Maill (M= 5.37, SD = 1.57). 

Findings confirmed studies in the literature for pronoun use (Kiesler & Sproull, 
1986), response extremity (Booth-Kewley et ah, 1992; Dillman et ah, 2001; Helgeson & 
Ursic, 1989; Kiesler & Sproull, 1986; Naemi, 2006; Petit, 2002), response equivalency 
(Bachmann et ah, 1996; Booth-Kewley, et ah, 1992; Hayslett & Wildemuth, 2004; 
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Helgeson & Ursic, 1989; Kiesler & Sproull, 1986; Mehta & Sividas, 1995; Pettit, 2002), 
yea-saying (Kiesler & Sproull, 1986; Weijters et al., 2004), and item completion errors 
(Kiesler & Sproull, 1986; Pealer, 1999; Smee & Brennan, 2000). Given the current 
study’s findings in the context of other studies, one may expect no survey mode 
differences in pronoun use, response extremity, yea-saying, item completion errors, and 
response equivalency in studies with similar populations and circumstances as the current 
study. Additionally, one may expect no response quality differences among early and late 
responders. 

Findings regarding item completion errors were not significantly different by 
mode or wave; however, their presence in all response conditions may still reduce data 
quality. Results of the question 10 analysis for this variable revealed that 52 of 880 
respondents (6%) provided a mean of 8.98 (SD = 13.35) responses that should not have 
been included in data analysis based on their response to question 10. Results of the 
question 14 analysis revealed that 1 12 of 880 respondents (13%) provided a mean of 3.54 
responses that would not have been included in data analysis if respondents had correctly 
followed branching and skip instructions. The presence of a substantial number of 
responses where they should not exist may influence a study’s findings. Applying rules to 
“clean” the data to remove invalid responses, however, may invoke ethical concerns. 
Reducing the problem before it occurs may be the best solution. Regardless of the survey 
mode, suggestions to reduce item completion errors include designing appropriate items 
and instructions, using cognitive interviewing to understand how respondents interpret 
items and instructions, and pilot-testing questionnaires (Dillman, 2000). 

Findings from supplemental analyses regarding anecdotal comments and multiple 
response use suggest that although no differences were observed by response wave, these 



151 




variables may add substantial value to understanding mode effects and interpreting survey 
findings. Thus, they are worthy of capturing and analyzing. Multiple response use, in 
particular, may influence comparisons between mail and web survey data. While web 
surveys can use radio buttons to prevent web respondents from providing multiple 
responses where they are not expected, mail surveys cannot. As a result, rules may need 
to be applied to mail survey data to determine which response to keep and which to 
disregard. Applying such rules to “clean” the data may influence or change results. 
Additionally, applying such rules to one mode and not to the other reduces comparability 
and equivalence between the two. Future studies comparing parallel mail and web modes 
should include ways to capture and analyze anecdotal comments and multiple response 
use in web modes where possible. In the web mode, text boxes may be strategically 
placed throughout the questionnaire to capture anecdotal comments. In the mail mode, 
clearer directions and better item structuring may reduce multiple response use. 

Findings regarding item nonresponse contradicted studies showing the same 
(Pealer, 1999) or lower item nonresponse in web versus mail surveys (Kerwin et al., 

2006; Kiesler & Sproull, 1986; MacElroy, Mikuski, & McDowell, 2002; Schaefer & 
Dillman, 1998; Stanton, 1998; Truell et al., 2002); however, supported three studies that 
observed higher item nonresponse in web surveys (Ahlstrom, 2004; Smee & Brennan, 
2000; Jun 2005). It is assumed that the inconsistent findings for this variable are due in 
part to differences in the variety of automated error-checking features used in the web 
survey mode. For example, Kerwin et al. (2006) observed substantially lower item 
nonresponse in the web versus the mail survey because their web version included an 
automated feature that infonned respondents of the number of incomplete items. 
Respondents were then given the option to return to complete those items before 
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submitting the survey. Although some of the studies reporting lower item nonresponse in 
the web surveys suggest that their mail and web surveys were identical, they provide little 
or no infonnation about the automated features used in the web mode for a reader to 
make a valid comparison (e.g., Kiesler & Sproull, 1986; MacElroy, Mikuski, & 
McDowell, 2002; Truell et ah, 2002). 

In contrast, Jun (2005) reported higher item nonresponse in the web survey, noting 
that a substantial number of dropouts were observed among web respondents after 
completing the first page, whereas mail respondents who did not complete their surveys 
were not likely to mail them in. This study used no automated features in the web mode 
to ensure that the mail and web formats were as parallel as possible. Smee and Brennan 
(2000) conducted a survey using the following parallel mail and web formats: mail, a 
single web page with no automated features, a multi-page web survey with automated 
branching and no error-checking, and a multi-page web survey with automated branching 
and error-checking requiring respondents to fix their response errors. They observed more 
item nonresponse in all of the web modes compared to the mail mode, noting that it 
appeared to be easier to abandon a web versus a mail survey. They also observed that the 
use of the multi-page web survey with automated branching and response validation 
produced the highest number of partially completed questionnaires compared to the other 
formats. The latter fonnat prolonged a respondent’s time to complete the survey due to 
the error-checking features as well as the larger file size which required more time for the 
page to load. 

The web mode in the current study used no error-checking and minimal 
automated branching features. In addition, web respondents were assigned an 
identification number and counted as a respondent immediately upon accessing the 
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survey’s main page. Use of the final page’s “submit” button could not be tracked, so it’s 
possible that some web surveys were abandoned before the respondent clicked the 
“submit” button. Given the current study’s findings in the context of other studies, one 
may expect that item nonresponse will be higher in web surveys in studies with similar 
populations and circumstances as the current study when automated features are not used 
and possibly when automated features become burdensome. Future studies comparing 
parallel mail and web modes should carefully examine the influence of automated 
features on item nonresponse to advance the field’s understanding of this phenomenon. 
Such studies should also provide detailed descriptions of the automated features used so 
readers can make informed comparisons among studies. Future studies should also track 
use of the final “submit” button to distinguish completed versus abandoned web surveys. 

The current study’s finding of greater response length in web versus mail surveys 
is consistent with the literature (Kiesler & Sproull, 1986; MacElroy et ah, 2002; Nicholls 
et ah, 1997; Schaefer & Dillman, 1998; Wu, 1997). No studies were found that observed 
higher response length in mail versus web modes. Given the current study’s findings in 
the context of other studies, one may expect that response length will be higher in web 
versus mail surveys in studies with similar populations and circumstances as the current 
study. 

Research Question Three 

Research questions three was, “Do demographic characteristics and response 
quality predict survey mode or response wave?” Results revealed that increasing age, 
gender = female, shorter response length, and less yea-saying behavior predicted an 
increased likelihood that a respondent self-selected the mail survey mode. No variables 
predicted response wave. Researchers must carefully consider the effects of these 
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variables when designing mail and web surveys with mode self-selection. In studies 
involving mode self-selection, using a mail and web mixed-mode survey design may 
pennit a researcher to realize the added value of a web survey’s greater response length as 
well as the expanded demographic coverage that the two combined provide. 

Response Wave 

No differences were revealed among demographics or response quality by 
response wave. This finding supports research reporting no differences in a variety of 
demographic and response quality variables between early and late responders (Bostick et 
ah, 1992; Fraze, 1986; Gillispie, 1997; Goudy, 1976; Irani et ah, 2004; Sobal & Ferentz, 
1989), but contradicts one study reporting differences in demographics (Dallosso et ah, 
2003). This contradiction may be due to a topic’s relevance to specific groups in a 
population. Dallosso et al.’s (2003) study of incontinence and urinary symptoms among a 
randomly selected sample of 1,050 participants age 40 or older compared the 
demographics of early and late responders. Results revealed that women were less likely 
to be late responders (OR 0.88, 95% Cl 0.81-0.96), South Asians were more likely to be 
late responders than Caucasians (OR 1.92, 95% Cl 1.66-2.22), and the likelihood of late 
response decreased with age (OR 0.99, 95% Cl 0.98-0.99) (Dallosso et ah, 2003). 

In the current study, it was assumed that the survey topic (school counselor work 
settings, caseloads and activities) was equally relevant to all school counselors in the 
population. Findings suggest that there were no adverse effects produced among the study 
variables by carrying survey data collection into the beginning of the new school year. 

Implications 

Given the nonexperimental nature of the study design, external validity 
(generalizability) is limited; however, the study is important for the following purposes: 



155 




(a) Findings are valuable to infonn future studies for which time and resource constraints 
prevent the use of experimental designs; (b) Substantial details regarding methods, 
analysis, and web survey features used in the current study will help readers make 
informed comparisons with similar studies; (c) No mode differences in pronoun use, 
response extremity, yea-saying, item completion errors, or response equivalency, 
suggesting that response quality is similar among mail and web surveys for these 
characteristics; (d) Age and professional employment differences by mode suggest that 
the two modes are not representing the same populations on these variables. Thus, future 
web and mail surveys must be designed with sensitivity to age and professional 
employment demographics; and (e) Though the web survey had greater item nonresponse 
than the mail survey, it had greater response length. Research has shown that automated 
features in the web survey interface can be programmed to effectively reduce item 
nonresponse, thus overcoming this shortfall. Given the web survey’s comparability to 
mail surveys in pronoun use, response extremity, yea-saying, item completion errors, or 
response equivalency, it’s added value of producing greater response length, and its other 
benefits (speed, lower comparative costs and resources, automated data entry), web 
surveys can be very advantageous as a means to collect survey data alone or in a mixed 
mail-web design. 

Limitations 

Findings should be interpreted within the context of limitations regarding the 
study’s nonexperimental design, assumptions about respondents’ access to the web 
survey, and web management issues. Regarding the design, this study used a 
nonexperimental design with nonrandom sampling, survey mode self-selection, and no 
reliability analysis. These factors limit generalizability beyond the participants, setting, 
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survey mode, and other circumstances specific to this study. Regarding web access, initial 
contact inviting participation was delivered in the mail survey mode. Thus, web survey 
respondents had to take the extra initiative to locate a computer, connect to the Internet, 
and type in the URL address to access the web survey. It’s assumed that web respondents 
completed their surveys using computers with the appropriate hardware, software, 
settings, and Internet connectivity for the survey to load, display, function, and submit 
correctly. It’s also assumed that web respondents were sufficiently computer literate to 
locate the survey web site, navigate within a web page, provide responses using a 
keyboard, and submit the survey electronically. Regarding web survey management, web 
respondents were assigned an identification number and counted as a respondent 
immediately upon accessing the survey’s main page. Use of the final page’s “submit” 
button could not be tracked, so it’s possible that some web surveys were abandoned 
before the respondent clicked the “submit” button. 

Recommendations 

The purpose of this study was to examine differences in demographics and 
response quality between survey mode and response wave when mode self-selection is 
used. Its findings contribute new infonnation to the growing field of survey research and 
practice. 

Use an Experimental Design 

Further research should conduct a similar study using an experimental design. 
Such a design would include random sampling, random assignment of participants to 
groups (including a control group), random assignment of groups to conditions, and 
controlled extraneous variables. Experimental groups for the web survey mode could 
compare the web formats by varying the automated features such as those suggested by 



157 




Smee and Brennan (2000): a single web page with no automated features, a multi-page 
web survey with automated branching and no error-checking, and a multi-page web 
survey with automated branching and error-checking requiring respondents to fix their 
response errors. Conditions of respondent mode self-selection could also be varied as 
well as conditions of sequential versus simultaneous mail and web mixed-mode 
conditions. Switching modes in follow-up contacts (e.g., sequential method) has been 
shown to improve response rates over some simultaneous designs (Dillman et ah, 1984; 
Paxson, Dillman, & Tarnai, 1995; Schonlau et al., 2002; Shettle & Mooney, 1999). This 
research could be expanded to include an analysis of demographics and response quality 
under these circumstances. A reliability analysis should be included and the range of 
demographic and response quality variables should be expanded. 

Vary Response Wave Conditions 

The current study was comprised of an initial solicitation to participate followed 
by four reminders, but used only two response waves. A future study should be conducted 
to compare variables among early and late responders by creating a response wave the 
initial solicitation plus each reminder (Pavalko & Luttennan, 1973). 

Anecdotal Comments and Multiple Response Use 

The current study observed a difference in response length by survey mode, but an 
analysis of response length in anecdotal comments by survey mode was not possible. 
Additionally, findings indicated that response length in anecdotal comments, pronoun use 
in anecdotal comments, and multiple response use was each statistically different than 
zero, suggesting that may be valuable variables to capture and analyze. Future studies 
should extend these findings to analyze anecdotal comments and multiple response use in 
both mail and web survey modes. In the web mode, text boxes may be strategically placed 
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throughout the questionnaire to capture anecdotal comments. In the mail mode, clearer 
directions and better item structuring may reduce multiple response use. 

Response Quality 

Although item completion errors were not significantly different by survey mode 
or response wave, the presence of invalid responses in a dataset may reduce overall data 
quality and influence results. Future research should explore ways to reduce item 
completion errors in all survey modes, although special attention should be paid to the 
role of automated features in reducing item completion errors in web surveys. 
Additionally, future studies comparing parallel mail and web modes should examine the 
influence of various automated features on item nonresponse. An analysis tracking use of 
the final “submit” button to distinguish completed versus abandoned web surveys would 
be very helpful in advancing the field’s understanding of item nonresponse in web 
surveys. The current study examined differences in demographics and response quality by 
mode and wave; however, future studies should expand this analysis to examine the 
effects of demographics on response quality. Future studies could also analyze the quality 
of open-ended responses with respect to response sophistication, reading level analysis, 
and unique themes. 

Cognitive Interviewing 

Schonlau et al. (2002) argue, “little is known about the effects of web survey 
instrument design on how survey participants respond to a particular survey question or 
the survey as a whole, or what sort of design enhances response rates or information 
accuracy” (p. 79). An analysis comparing mail and web surveys using cognitive 
interviewing and related methods can address this gap as well as detennine respondents’ 
item comprehension, explore what respondents consider when selecting a response, 
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identify item ordering effects and response burden, and understand why respondents skip 
items (Sirken et al., 1999; Sudman et al., 1996; Tourangeau et ak, 2000). 

A Theory of Mail and Web Survey Design Issues 

A review of the literature comparing and contrasting mail and web surveys 
provides a range of studies addressing quantitative features such as demographics, 
response rates, costs, response quality, social desirability bias, etc. While many of these 
studies produce somewhat consistent findings for certain variables by survey mode (e.g., 
age, ethnicity, pronoun use, yea-saying, and response length), some of them do not (e.g., 
gender, item nonresponse and item completion errors), making it difficult for survey 
researchers and practitioners to develop a set of expectations that can be depended upon 
when designing future studies. The inconsistency of some findings may be due to the 
influence of design issues between the two modes within studies (e.g., how the survey is 
received, different reading and writing formats, respondent burden, mode completion 
time, visual display), as well as the variety of automated features available for use 
between web surveys in different studies. Very few studies were found that examined 
design issues between mail and web modes (see Jones, Fraser, & Dowling, 2005) and no 
studies were found proposing a theory of how such design differences influence 
demographics and response quality. It has been suggested that potential differences in 
respondent burden associated with differences in questionnaire length and the time to 
complete (and download if a web survey) a questionnaire (Couper et al., 1997; Fisher & 
Kydoniefs, 2001), differences in perceived stress associated with answering sensitive 
items, functionality, and task demands may contribute to mail and web differences (Fisher 
& Kydoniefs, 2001; Jones et al., 2005). Differences in the visual display of questionnaires 
may also produce differences in mail and web modes (Dillman, 2000). While a mail 
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respondent sees a questionnaire in the exact format the researcher intended, web surveys 
may display differently or even incorrectly depending on variations in a respondent’s 
hardware and software (Dillman, 2000). Tourangeau (2003) comments, 

We are only just beginning to have a sense of. . .the key variables that determine 
whether there is agreement or disagreement across modes of data collection. The 
issues raised by web surveys are particularly hot right now partly because web 
surveys are primarily visual and use a much wider range of visual material. . .than 
has been true of surveys in the past. (pp. 5-6) 

Future studies addressing gaps in the field’s understanding of circumstances that 
contribute to similarities and differences in mail and web survey modes would do much 
to advance the practice of survey research. 
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Career Connections Research Center 
Creating Pathways to IT Careers Project 
Creating Career Connections for Students with Disabilities Project 



September 26, 2005 
Dear School Principal: 

We are writing regarding survey materials we mailed you late last spring. As we indicated then, 
we are collaborating with the Illinois State Board of Education (ISBE) to study career development 
activities in Illinois schools. Funded by the National Science Foundation and U. S. Department of 
Education, this research includes surveys of school counselors and IT-related CTE teachers to learn 
about their involvement in assessment, planning, and recruitment activities with students in grades 6 - 
12 . 



Your assistance is absolutely essential to this research! However, we have not yet heard 
from you or your staff. Enclosed is another set of survey materials. We need your help to distribute the 
materials to the relevant counselors and teachers in your school. With their input, we will gather new 
information about counselor and teacher involvement in specific career development activities. 

Please be assured that everyone’s participation is strictly voluntary. Also — be assured that the 
information you provide is completely confidential, as are the responses of your staff. We will not 
share the information with your colleagues, administration, district, state, or anyone else. 



We ask that you help with two things: 

• Review the enclosed Principal’s Survey Distribution Record. On it, we list the names of the school 
counselors and CTE IT- related teachers identified for your school. Please correct this list and/ or 
add new names as needed. Then return the corrected copy to us in the envelope provided by 
October 14. 

* Distribute the enclosed survey packets right away to each individual on the corrected list so they 
can respond by October 21. Please call or email ns if you need additional packets. Counselors and teachers 
can use the enclosed questionnaire or complete the survey on-line— whatever is easiest for them. It 
should take less than 30 minutes to complete the survey. 



Please call us right away at 269-387-6181 if you have any concerns or questions. Again — forthis 
research to be meaningful, we need your assistance. With your help and the participation of your 
counselors and teachers, we will gather new knowledge about career assessment, planning, and 
recruitment activities with students in grades 6 - 12. Thank you for your time! 



Respectfully yours , 






Paula D. Kohler, Ph.D. 

Professor and Project Director, paula.kohlei@wTnich.edu 



3506 Sangren Hall Kalamazoo Ml 49008-5258 
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Illinois State Board of Education 



100 North First Street • Springfield, Illinois 62777-0001 Jesse H. Ruiz 

www.isce.net Chairman 



Rod Blagojevich Dr. Randy J. Dunn 

Governor State Superintendent of Education (interim) 



MEMORANDUM 



TO: 


Illinois School Principals 


FROM: 


Mark Williams, Division Administrator^*^ 
Career Development and Preparation 


DATE: 


April 21, 2005 


RE: 


Support of Research 



The Career Development and Preparation Division of the Illinois State Board of Education 
endorses the federal research of 1) Creating Career Connections for Students with 
Disabilities: A Longitudinal Study of their Enrollment and Outcomes of Career and 
Technical Education and 2) Creating IT Career Pathways through High School Career and 
Technical Education Programs. The success of this study is a collaborative effort, and it is 
important for you to encourage your teachers to participate. 

Thank you for your assistance with distributing the survey packets and returning the distribution 
record to Dr. Paula Kohler and Dr. Brooks Applegate at Western Michigan University. Your 
participation and encouragement will be highly valued and appreciated! 

The contact person in the Career Development and Preparation Division is Carol Brooks and 
she can be reached by e-mail at cbrooks@isbe.net or by phone at 21 7/786-4620. Thanks for 
your support. 



Printed by AF_-CiO (AFSCME Local #2311 and 1FSOE Local #3236) Employees 
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/ Western Michigan University 

Creating IT Career Pathways Project 
and Creating Career Connections Project 


Principal's Distribution List 




Survey 

Distributed 


If survey not distributed to the 
individual listed, please indicate 
action taken: 


Name 


Position 


Yes 


No 


Returned 
to WMU 


Given to (Enter Name) 










































































































































































































































































































































































Please complete this form as soon as possible 
and return to WMU ill envelope provided. Thank you! 
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APPENDIX C 

PACKET FOR SURVEY PARTICIPANTS 




1 INIVFR.S1TY 

College of Education 
Career Connections Research Center 
Creating Pathways to IT Careers Project 
Creating Career Connections for Students with Disabilities Project 



September 26, 2005 



Dear School Counselor: 



We arc exploring career development activities in Illinois schools through two federally-funded 
projects sponsored bv the National Science Foundation and I J.S. Department of Education — Office of 
Special Education Programs. This research includes a survey of school counselors and CTE-IT teachers 
to learn about your involvement in assessment, planning, and recruitment activities with snidents in 
grades 6—12. 

We selected you to parUcipate in our survey because you have school counseling responsibilities 
with students within grades 6 — 12. Your participation is very important to us! With your input, we 
will gather new information about counselor and teacher involvement in specific career development 
activities. 

Y'ou can participate in this study — which should take less than 30 minutes — in one of two 
ways. Choose the way that is easiest for you: 

* Complete the enclosed survey and return it directly to us by October 21 in the envelope provided 
(do not return to your principal), or 

> Complete the survey on line by October 21 at: http:// www.su rveyz.com /TakeSurvey?id=23 228 

Your response is completely confidential. As soon as we receive it, we will check your name 
off our participant list, cut it from the paper survey or electronic file, and assign a code number. Only 
the researchers have access to code numbers. We will not reference you as an individual respondent 
and we will not share your responses with your colleagues, administration, distnet, state, or anyone else. 
(If you decide not to participate, please return the unanswered survey in the enclosed envelope and wc 
will remove your name from our participant list.) 

Please call us right away at 269-387-6181 if you have any questions or concerns. Again, your 
participation in this survey is important! We believe that your experiences as a school counselor are 
valuable and that your perspective will aid us in our exploration of career assessment, planning, and 
recruitment activities with students in grades 6-12. Thank you in advance for your participation! 




Paula D. Kohler, Ph.D. 
Professor and Project Director 
paula.kohler@wmich.edu 



3302 E Sangten Hall. Kalamanx. Ml 49008-525* 



Western Michigan University 



Dear School Counselor: 

We are collaborating with the Illinois State Board of Education (ISBE) through two 
federally-funded projects to explore career development activities in Illinois schools. The Creating 
Pathways to IT Careers Project (funded by the Nauonal Science Foundation) is investigating factors that 
influence enrollment, concentration, and employment experiences of high school females in Career 
and Technical Education-Information Technology (CTE-IT) programs. The Creating Career 
Connections Project (funded by the U.S. Department of Education — Office of Special Education 
Programs) is investigating factors that influence enrollment, program concentration, and 
employment experiences of high school students with disabilities in Career and Technical Education 
(CTE) programs. Both of these projects include a survey of school counselors and CTE-IT teachers 
to learn about your involvement in assessment, planning, and recruitment activities with various 
groups of middle and high school students. 

We have selected you to participate in our studies because you serve as a school counselor in 
Illinois. Your participation is very important to us! We know your time is valuable and that our 
request that you spend time on the survey may be an inconvenience. But, with your input, we will 
gather new information about counselor and teacher involvement in career assessment, planning, 
and recruitment activities. Your participation — which consists of completing a survey — is strictly 
voluntary. To participate in our survey, which should take about an hour to complete, please read 
die enclosed instructions. You can use the enclosed questionnaire to respond or complete the survey 
on-line — whatever is easiest for you. In cither case, you may choose to not answer any question and 
simply leave it blank, or to not participate at all. If you choose to not participate, please return the 
unanswered survey in the enclosed stamped envelope and we will remove your name from our 
participant list. 

All responses will be reviewed only by our research team and will be kept completely 
confidential. When we receive your response — either through the questionnaire or on-line — we 
will assign it a code number and your name will be removed from the response. Only the researchers 
will have access to coding information and no reference will be made to you as an individual 
respondent. We will not share your responses with your school, district, state, or anyone else. The 
on line survey responses will be collected through SurvcyZ.com, a system that maintains the data 
behind a firewall, accessible only to the researchers. All aspects of the data are keyed to our account 
and cannot be accessed by anyone else. 




College of Education 
Career Connections Research Center 
Creating Pathways to IT Careers Project 
Creating Career Connections for Students with Disabilities Project 



Western Michigan University 

H. S. I. R. B. 

Approved (or use lor one year from this' date: 



APR 0 5 2005 




3506 Sangren Hall, Kalamaroo, Ml 49008-5258 
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School Counselors 
Page 2 



If you have any questions about this survey or our studies, please feel free to contact either 
of us at the numbers listed below. You may also contact the Chair of the WMU Human Subjects 
Institutional Review Board at 269-387-8293 or WMU’s Office of the Vice President for Research at 
269-387-8298 if you have any questions or if any problems arise during the course of the study. For 
more information about our projects, visit our web site at www.wmich.edu/ ccrc. 

Again, your participation in this survey is important] We believe that your experiences as a 
school counselor are valuable and that your perspective will aid us in our exploration of career 
assessment, planning, and recruitment activities with students in secondary education. Your 
participation will help us generate new knowledge important to our field. Thank you in advance 
for your time and input! 



Respectfully yours, 

Paula D. Kohler, Ph D. 
Research Investigator 
269-387-5955 

paula.kohler@wmich.edu 




Brooks Applegate, Ph.D. 
Research Investigator 
269-387-3886 

brooks.applegate@wmich.edu 



Please Note 

This invitation to participate in our survey has been approved for use for one year by the WMU 
Human Subjects Institutional Review Board (HSIRB) as indicated by the stamped date and signature 
of the board chair in the upper right comer. Do not participate in this study if the stamped date is 
older than one year. 
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School Counselor Survey 
Career Connections Research Center 
Western Michigan University 
Instructions 

1 Two formats of this survey are available for your convenience: 

a. Record your responses on the enclosed questionnaire and return it 

directly to us in the envelope provided (do not return to your principal), or 

b Complete the survey on-line. To respond on-line, connect to: 

http://www.surveyz.com/TakeSurvey ?id=23228 

When you take the survey on-line, you may stop at any time, save your 
responses, and then complete the survey later Specific directions for 
doing so are provided at the site Your responses are not recorded until 
you click the submit button. 

2. This survey is designed to gather information about how you are involved in 
career assessment planning, and recruitment activities with the students for 
whom you are assigned responsibility (i.e , your “caseload" of students). 

3. When responding, please answer the questions in the context of the 2004 - 
2005 school year. 

4. Some items can be answered simply by checking one of several responses. 
Other items allow you to select more than one response. Still other items 
allow you the opportunity to provide your own brief description On each item, 
we indicate your options for responding. 

5. There are no right or wrong answers! Please respond to all items in a way 
that is accurate and appropriate for your particular situation 

6. If you have questions, please call or email: 

Shirley Swift: 269-387-6181. shirley.swift@wmich.edu 

Dr. Paula Kohler: 269-387-5955, paula.kohler@wmich edu 

Dr. Brooks Applegate: 269-387-3886. brooks applegate@wmich edu 

7. Please complete the questionnaire by October 21, 2005! 



Thank you for your participation ~ we value your input! 
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APPENDIX D 
MAIL SURVEY 



C* Ml la>Mtu>> . wM'p 



Career Connections Research Center 

Western Michigan University 

Phone: 2SS-387-S181 
Fax: £69-387-2826 



School Counselor Survey 



Spring 2005 



You may take the survey on-line at: 
lh tt p : i’.’WW w .5 u rv e vz.co m 'Ta k e S u rve v? id =2 3 2 2 8 
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SECTION 1 

School Counselor Work Setting and Caseload 



1. Are ycj :he Individual wtiaw name appears on (ha abe In the right corner rtf Ihe cover? 

□ Yes □ Nc - ITnc cl=a&= cress cul Lh= mn= letea -aid write year raTa 

2 . Do you work aa a school counselor tti la y aar w 1h si. ids nte In any grade 6 ttinough 12? 

□ Yes □ Nc - STOP and return ire su"vey h me envelope prcvkled 

3. Are you s 35 g nod tc worh with r.ut ante In one a peeing ashool or m«e than one eon col ? 

□ One setod O More Iran ore Ectod 

A. Please provide the school nameis| and thB percent of your work time all coated 1o eaoh eon col: 



School name 



Percent or your work lime 



-25^ 


2G-50W 


51 - 7514 


76-inC-% 


□ 


□ 


O 


II 


II 


□ 


□ 


II 


□ 


□ 


□ 


II 


c 


□ 


□ 


C 


II 


□ 


O 


II 



IF you work hi morelhan one school, c ease answer me remaining quest ; is based cn me 
echoDl al which you epend me moet time, ir y-ou spend equal lime al tj t c e schools, plch 
one cn whlcfl to base your responses and enter It below: 



The following questions refer to your “caseload 1 ’ of students. 
By caseload, we mean the students for whom you are 
assigned specific responsibility. 

5. Does you r case load consist -cf siY the students In your school or a a retl'lc group 
or groups or students? (Cheek only one;i 

□ A sruderts h the school ;G*p1o question 5) □ Specrc ^rcuc. Ej of slLd=nlE 



1 
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6j 7. If you ndlcated above that your case cad Is c ;mcr 3 ed oTspeciric group# dT otudsnts. please 
check a!! group ceflnltlor? Ihal apply and daacrlde the group. 

□ Spedllc graded 

Z) A chacellc crojp 

Q SlLdEnrta lr 3 spEdllc cutIcjIlti 

□ Dceo ai poguadcnis] 

B. t :i t perrantaganf VDiir tine c a you spend (1J working clrect y with etudents and |£| on al 
ills other taaka or sot vine 3 where you are not working directly with etunente? 

Una vrahJng drEc:fywlth Eluaeils ^ lln= not wart ng alrEcUy wtlh stud ml b 

3. How many etudante arson your total caseload this school year? 

ENTER. N L. USER 

id. OT the atudente on your tolal caseload this a chool year. how many receive s pedal 
education servlcea? 

ENTER NUMBER Of sp«aed BlLdSTllB on 035=1030 

1 1 . in whal grades are the stjdenls or your caseload? 1 Chech al I that applyj 

12. Please check how often yau will meet Indlvldua ly with the sludenla on your caseload during 
Inis s chool year: 

Q*er 4 

0-2 times 3-4 tlmee tlmee NA 

Typical student wild a dteaWty □ C C C 

Typical student wlltidiit a -dlsaCllby □ C C C 

1 3 . Dur n 3 Ihls s chool year, please list the three adlvltles or tasks on which ycu spend the 
mcsttlme 



2 
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SECTION 2 

Career Assessment Activities 



The questions in this section will help us understand more about how 
formal career assessments are implemented in your school and school 
counselor involvement in implementing these assessments. By 
formal career assessment, we mean assessments or assessment batteries 
or ''packages" designed to provide information regarding students' 
career interests, aptitudes, abilities, or other constructs that help students 
make career choices and/or choose career paths. 



14. Ssnerally do students on your caseload participate In any career assessment^;. al some lime 
Jun j Ihe r seconds r y ec jos:lon ^ (Cheek onei 

□ Yes □ No (Skip to Section 3- sags 7) 

15. WTilcti students on your caBeload typ sally participate In career assess montisl aur n a Ihelr 
secondary education? JChacK only onei 

□ AJ s‘uder:s (Skip to C jsstion 1 7) 

□ Only specnic z roups :r s: Jd=r:& i = g„ diose lr specific career iracts a "ic.ior pocu allot c "ojfs: 

16 Please Indicate. *vhlcn proup(si of sludenta on your caseload participate n career 
assesement|s|. | Check all that apply: 

□ College crsp etde'ile 

□ Ganeer anc IecTi ed iCTEJiortech p r epslLd=nl5 

□ DlLdinle who receve epeclal edLoaUDfi services- 

□ 3lLd=nlE wio a.'T? Ecoiorrlcally d sscvsrtasaa 

□ DlLdenlB who a -e acaaeTlca ly d fcachrartagec 

□ 3iLd=niE with in led Ergiiei pranciency 

□ 0:he-.;t. 



Questions 17 through 23 focus on students with and without 
disabilities on your caseload. If your caseload does not include a 
specific group of students, please indicate HA. 

Students with disabilities are students who receive special 
education services. 



■5 
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17 . In general. what ptr:;en:aqt oT Hie atudanta on your caaeload participate In career 
as-aesament|s| du ring their secondary education? 



Sfcderte wllh dtsat 4 ini=E 
SfcdentB Pinout diaab idee 

1 B. How often doea the typical aludent on you r caaeload participate In canear aasesament during 
his/her secon dary education? 



Nona 

□ 

□ 



l- 

25% 

□ 

□ 






□ 

□ 



Sl- 

75 E -s 

□ 

□ 



.t- 

1 QD% 

□ 

□ 



iflA 

□ 

□ 



SlL d=rriB w rtn d sat- IMes 
S lLdente without clsael :es 



Never 



□ 

□ 



Onoa, Annually 
Ones- bul not or more 
annually often 



□ □ □ 

□ o □ 



HA 



□ 

□ 



1 9 . Typically, at what grade do the atudenls on your -cas-a oad Tirat pa rllclpale In career 
aiaessTent? 



Students wlfhdl&abirtles. 

Students without 
alsEb Idea 



Hons 


£ 


7 


B 


5 


10 


11 


12 


HA 


□ 


□ 


o 


□ 


□ 


□ 


□ 


□ 


□ 


□ 


□ 


□ 


□ 


□ 


□ 


□ 


□ 


D 



other 

Ispeciy; 



20 . ■".■here do the stu dents on ycur caseload nrsl part clpats In a career aaseasment? 





Nowhere 


Area 

VbC 

canter 


Your 

school 


Another 

school 


NA 


Other 

ispactryj 


itudEfits with al&eb lilts- 


□ 


o 


□ 


□ 


□ . 




tudenls without dsacIMaa 


□ 


□ 


□ 


□ 


3 . 
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21 . In your school . who has- primary 
students on your case pad? 



Student wiiti dsasiinteE 
SfcKfertewlifiBiitdleab Ills* 

22. ir anothar person has pr Tarv responsible sty Tor Imp emending career aBS.BsameH(B;i for the 
b:l cants win dlea a Itles on ycur -caeelbad. please r d Icale that parson : 

□ Another schao conEeo r .;Ej □ School pByctolags::si 

□ AcadeTlctadier(Bj □ District acmlnfctralor 

□ Career and technical Educational [CTEJteacftenjs] □ Gchco acmlrlstrafor 

□ Special edJcattnleadnEfiBj □ ether release desclt-a): 



23. if another peraon has primary responelbll Ity Tor Implementing career aasos amentfe} for tha 
atuflents without disabilities on your caseload. plaaae indicate thal peraon: 

□ Another schao ooi*nBElon;E;i □ School pEyctologls.:i&] 

□ AcaderrlcTeacnens* □ District acmlrlstrsfor 

□ CareEf and tectmleal EdiscaUDnal (CTE)iEacter.» □ Schsol actilrisJtrstor 

□ Coeds eaucaton Tescneos. □ Ctner release descrlbej: 



rsapsnsltll ty for Implementing career aeaeasmant|S;Torthe 

Wa one I co MA 

petj-.r 

c □ □ c 

cone 



2ri. 



tthat percenters of the students w ih disabilities on your caseload typically participate In each 
of Ihe following type® of assessments? IT your caseload does not Includs studenls with 



disabilities, please chacH.HA. 



iNone 



EjeIo aptitudes 


c 


Cs^eer nte p e&l 


c 


Persoralty pro" e 


c 


'A'crt vauas 


c 


'A'crk adjustment 


c 


PnyBlcal fjictonlrg 


c 


Ctiet ;p ease enrier type; 


c 


Chet pease ante - type; 


□ 



i- 

25% 


26- 

50% 


51 - 
75% 


76- 

10<]% 


HA 


□ 


□ 


□ 


□ 


□ 


□ 


□ 


□ 


□ 


□ 


□ 


□ 


D 


o 


□ 


o 


□ 


O 


□ 


□ 


□ 


□ 


□ 


□ 


□ 


□ 


□ 


□ 


□ 


□ 


□ 


□ 


□ 


□ 


□ 


□ 


□ 


D 


□ 


□ 
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What percentage or the s Ij-de Tla wlthoul disabilities or, your cae-Blosd typical y participate In 
each oT the fc- low ng types of assessment 7 ir your caeslaad does r cl I nclude students 
without dlsadll tea . please c neck HA. 





None 


H - 
35% 


36 - 
5(1% 


51 - 
75% 


76- 

ioa% 


NA 


3aslc act Usee 


□ 


C 


□ 


□ 


□ 


□ 


Career lr:=r=st 


c 


c 


□ 


□ 


□ 


□ 


=erEDnaiitypranie 


□ 


C 


□ 


D 


□ 


□ 


Wort values 


□ 


C 


□ 


□ 


□ 


□ 


Wort ac|uslrrer: 


c 


C 


□ 


□ 


□ 


□ 


=r>sicai 'unciicn ng 


c 


□ 


□ 


□ 


D 


□ 


Other ipleasa enta' type 




□ 


□ 


□ 


□ 


□ 


□ 


Other ipleasa enter type 




c 


c 


□ 


□ 


□ 


□ 


Please indicate if you do the following activities * th students on ycur caseload (shack all lhal 


apply.: 
















Students wflti Disabilities 


Students without ciesbiiiiiee 




Yes 


No 


NA 


yes 


No 


NA 


DeIoc: career asses&Tentis) 


□ 


□ 


c 


C 


□ 


□ 


AanhlELe r career assessments • 


□ 


□ 


□ 


c 


□ 


□ 


tnta r orat career assassnetlE 


□ 


□ 


c 


c 


□ 


□ 


Tor aludanta with Disabilities on your caseload, please 
and the percentage of sludente assessed with each: 


list up to 5 career assessments 
% ofatjcerts with o satillhleE 




Name of Assessment 






None 


i - 
25% 


2S - 


at - 

75% 


uvsr 

75 % 


1. 




□ 


D 


□ 


□ 


□ 


2. 




□ 


□ 


□ 


□ 


□ 


3. 




’ □ 


□ 


□ 


O 


□ 


4. 




□ 


□ 


□ 


□ 


□ 


5. 




□ 


□ 


□ 


□ 


□ 



6 



203 




2B F-^r students v.'ltn aut dls-abll Mob -a-n your coaoload, please list up to 5 c-aresr OBS-Bsamenls- and 
trie- psncBn:ago or atuitents asgossBd with each: 



% of Bluaoils without dfeablHeE 

Name or AsBoasma-nt 





Hona- 


1 - 
25% 


Li - 

50% 


51- 

75% 


Ovor 

75% 


1. 


□ 


□ 


c 


□ 


□ 


2 


□ 


O 


c 


□ 


□ 


o 


□ 


□ 


c 


□ 


□ 


4 


□ 


□ 


□ 


□ 


□ 


s. 


□ 


□ 


□ 


□ 


□ 



SECTION 3 

Career Planning Activities 

The next questions will help us understand the time you spend and hinds 
of activities you are involved in regarding career planning with the students 
on your caseload, Questions 29 through 31 focus on students with and 
without disabilities. If your caseload does not include a specific group of 

students, 

please indicate NA. 



204 




23 . For sach of the Tall ;w r g plsaeo Indicate- Ihs prop 5 1 on c J s-tudcnlc or your cawlaad -vlih 
-vhoi'i you Typically implement or Include in tti^ activity. 



G:joenfl&wUh c saclltleE 

None 1- 2fi- SI- 76- HA 

25 % SOVa 75 % IDOtt 


Nona 


siLd 

1 - 
25 % 


snlE without c saol tes 

26 - 51- 76- 

50 % 75 % 1DCt% 


NA 


flrtm ilE-1e'oa r -2&'at6=Esnen1i>;. 

□ □□□□□ 


D 


□ 


□ 


O 


□ 


□ 


Frm'de iTTcrmaHon negi'toing to&jHe of caTe=r 
a&&=ssn=nlE 

□ □□□□□ 


□ 


c 


□ 


□ 


□ 


□ 


Assist ElLdenlE to Icendiy carae r goals 
□ □□□□□ 


□ 


c 


□ 


□ 


□ 


c 


PiOh'de irtbrmadon at-ojt career areas ana-o' 

OCdOTE 

□ □□□□□ 


□ 


c 


□ 


□ 


□ 


□ 


Provide: moTiatoT atoat oar=e r ■equirsTie'ils 
□ □□□□□ 


□ 


c 


□ 


□ 


□ 


c 


Rnov de -3BEI&1S ice :: dEvc op cr mad y an 
eauoadans :> r c c ~ m o ar aligned tflfi career 
gott 

□ □□□□□ 


□ 


□ 


□ 


□ 


□ 


□ 


RnocdelmToTiaton acoat secondary oar=e r 
andlecmlca ecucaoon prac'aTS 

□ □□□□□ 


□i 


□ 


□ 


□ 


□ 


c 


RinKB irttermadon aCoat ac<s:-&e;3n3a'y 
ca’eer aic lEtflnlca acjoaton orotrams 

□ □□□□□ 


□ 


c 


□ 


□ 


□ 


□ 


Provide inlOnnadon aco at pod^eoanday 
eauoadonaJ InslllLions -le.g. . colleges ard 
UTlIve-^EtllEE | 

□ □□□□□ 


□ 


r 


□ 


□ 


□ 


□ 


least aarao r andtor emptoyaniry eki s 
□ □□□□□ 


□ 


c 


□ 


□ 


□ 


□ 
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3D For each of the rollon Irvg carear planning acllvltlBS. pleasa I ndicala tne context In ^hlcti you 
typically Implement the activity * 1h tha atufcnts on your cnrinil. 



BLudeiie wim diEst mas 



BluaeUe without dsatll^B 



One-c-n-Ona Group iHA 



Ona-on-Ona 



Adm Ts:ef ca r ear assassn=nfl;s] 



□ □ □ 

RTo-dE intomaton regard ng ■esjue o' cateEr 
assaEEn=nc& 

COO 

Asset &Hd=ntE So ktennry career goals 

c □ □ 

PTff- d= Informal: o"i acouLcarEe'a'eas ard or 
cpUcnE 

c □ □ 

Poo.lds InTormat ot about laneer reqL rEnails 

c □ □ 

P‘o- da assl&laioE :a develop or mcciry an 
educaUcnal p’ocran par all^nac filth cs^ear 
gos s 

COD 

Pto* da nfomatoi about secondary career and 
technical education crog p am& 

□ □ □ 

Provide nfornatci acoaL ocs:-ee;:rdi7 
caeer andlectiilca eaucaton procrsTS 

□ □ II 

Proves nromatoa acojL ao&:-Ete&cnda.7 
edLcallcial Insllutons ;e 3 ccIlEgES and 
uik'erElllEsi 

□ □ 

Teact carEe'ar i'o-'aTafcyaclIlty skits 

□ □ □ 



□ 

□ 

c 

c 

D 



□ 

□ 

I 



D 

□ 



Group 



c 

I 

□ 

(I 

□ 

I 

c 

I 

□ 

I 



HA 



□ 

□ 

□ 



□ 



□ 

□ 

□ 

□ 

□ 



□ 
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31. Please Indicate now Important It Is- that students on your case oad experience each oTIIne 
following. 



SlLd=ms with dfcaHIMes 

h cl Lcmewhal Very 

Important Important Important 


Students without dsatlim 

Not Somewhat 

Important Imporlanl 


EE 

Very 

Important 


=5Uclaa1eln cane=r ssseEBrenis i 

□ □ □ 


□ 


c 


□ 


^■EOElve nrormallan negmd no result; of careEr 
assessments 

□ □ □ 


□ 


c 


o 


^■KElve assistance n loertiy ng career go a Ss- 
□ □ II 


□ 


□ 


□ 


^ecalve nlPmarcn at-OLt car=e p areas ana or 
opoors 

□ □ □ 


□ 


□ 


□ 


^■KElve Informs: on at-OLt £aree p 
nequinEfnente 

□ □ □ 


□ 


D 


o 


^■EOEive assisianceio ae-eiap or modry an 
eo jcaional prograrr plan algned with caree p 
goals 

□ □ □ 


□ 


□ 


□ 


^■E0=lve nromsion at-OLt SEtoncary oaree p 
ard beann leal EdireaUtM arc^ p am& 

□ □ □ 


□ 


II 


□ 


^ecslve Saromsiion at-OLt cos1-saco'Kia p y 
caree'ard tesnnlcal Educal an crog p ars' 

□ J J 


□ 


II 


□ 


^eoElve Informaion atoLt posl-seGO'ida'y 
eGuca:lona heULltSI^hs ;-a.g., co eges ana 
universities-; 

□ □ □ 


□ 


c 


□ 
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SECTION 4 

Recruitment and Support for Students 
in Nontraditional Occupations 

The nest questions will help us understand how you might be involved 
in recruiting and/or supporting students in academic or career and 
technical (CTE) education programs considered nontraditional (NT) for 

their gender. 

Nontraditional (NT) programs are generally defined as those in which 
students of one gender represent 25% or less of those enrolled in the 
program {e.g., females in computer programming, males in nursing). 

32. r paner-al, t: what e *tsnt la NT enroll Ter t In curricula or progra t b emphaa zed In your 
bch-coi? iChecK one| 

□ NcH ac all □ Ssnawrat □sigrdcs'iUy 

For questions 3340, please indicate (I) whether you do the activity and 
(2) whether you consider the activity to be within the scope of your job. 



Review career planning naterlala end 


Do Ite atiferity 


Within scope o- J Jct- 


practices Tor: 


Yes 


NO 


Y65 


HO 


Senae-blas aiKMHrTepresemallon" 


□ 


D 


c 


D 


Racial w ethnic bias aic-tir Tecraseilallcn’ 


□ 


□ 


□ 


C 


Review curriculum and Instructional 


Do Its acct.1t}' 


Within scose o J Jet- 


materials for: 


Yes 


ho 


Y65 


NO 


Senaer olas jrdorT=pTeB=mator 


□ 


□ 


c 


C 


Fvaclal o'eimis tas aic-ior Yecrssenlallan' 


D 


□ 


c 


c 


Provide apecllle Information about non- 


Do ire accr.1t)' 


VWllll BBEXMS DtfJOb 
1 r J 


tndltlora careers srrd.'or programs Tor: 


Yes 


ho 


V65 


No 


CluaenLs 


□ 


(= 


C 


c 


PanentE-'Ouindtars 


O 


□ 


c 


D 


Ec jesters 


D 


c 


c 


C 


Acnlrlslrsijrs 


□ 


C 


C 


□ 
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3£. Con duct pr-c-Tesa lonal development 

GEncef andlH etmic bias 
hT careere andi'cr uroo'aTis 
Eeocjs na'sssTie'iL orevenUon 

37. Actively recruit: 

Females Id coLrses andfar programs n Bne 
ETE M areas -sclenae leenro icy 
englreerlng, madij 

Bbuaenls J SrCTE prec *Tis<ncn-TadlU;rial 
Tor KSnelr gerder 



32 . Provide role models an d.'sr mentora Tor: 

Finales =rmo ea in ire STEh-' areas 

Etuaenls ernoled In CTE programs ran- 
IradGona Tor tnelr gerder 



39. Provide support groups, peer counseling, 
or other support strategies Tor: 

Females erroled In ITe STEW areas 

Eluaenls erroled lr CTE programs rors- 
Iradtona Tor tnelr gerder 



iC Identity: 

ETEM area wort apoolunltles h me 
oammurltyTcr female sluders. 

har: p aalUo-i5l ■aoIc opportunities In Ihe 
ccnn jrlty Tor sUoents 



Do ITe adhiy 


vmm scope D*jcb 


VES 


ho 


-es 


ho 


□ 


□ 


□ 


r 


□ 


□ 


□ 


□ 


O 


□ 


□ 


c 


jo ire acthiy 


Wthln scope c J Jab 


VES 


ho 


-es 


ho 


□ 


□ 


□ 


C 


□ 


□ 


□ 


C 


Co ire aclU'rv 

■ 1 


WMiln scope o J Jab 


-es 


ho 


Yes 


ho 


□ 


□ 


□ 


□ 


□ 


□ 


□ 


c 


jo ITe aclfriy 


ViThln scope o J Jab 


VES 


ho 


-e& 


ho 


□ 


□ 


□ 


c 


□ 


□ 


□ 


L 


jo ITe acllvty 


wrhln saope ctf Jot? 


Yes 


ho 


-es 


ho 


□ 


□ 


□ 


c 


o 


□ 


□ 


c 
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SECTION 5 

Demographic Inform alion 

The following questions will tell us a little about your background. In 
studies like these, we like to compare the experiences and perspectives 
of people of varying backgrounds. 



4t. What Is your gender? Ferrate j Male 



43. What Is your racial c-r slhnlc background? 

1“ Black or .Africa i ArnEflcajn 
Q American Indian zr A aekan haHve 

□ Aslan 

□ Vihte cr CaLoaelar 



O LaUroHlssatld 

□ Naive Hawa la n o' olh=r FasHc lalanaer 
O Biradal cr mldraclal 






What Is your age? 



(Enter numserof years; 



44. Pteaee Indicate haw we I you think your forma! a ducatlon (u nderpraduale and graduate;- a nd 
son nu ng educallon ]lnsHr.!ce train! ngi prepared you to do thaae things 



FORMAL Education 



CONTINUING EdLOaHm 



Very well Moderately Not a: all Vary well 

WBIJ 

Meet the overall demands of ycu - oc 

□ □ □ c 

CondJcLoaree'aBBESsneTls and planr ng 
with EtiJdEfits wltt alaab Hies 

□ □ □ C 

Cniduct career aeEessmenlE and planrln-g 
with students without d saalltteE 

none 

Cor dLc; z-areer assessments ard parnlng WtSn students with 
roclal o' ethnic tacks rojrds d'-erenrt from roLra 

□ □ □ c 

Use specif; strategies 1c recruit slLdents to ounlEJlaar d or 
crocams nonlrad conal Tor their cerd=r 

nunc 

Use Ep-e-ofT ; strategies lo eupport sUdanls lr oufflcL a sns.'zr 
arcs rams nonlrad conal To - tnter cerdar 

□ □ □ □ 



Moderately Not at si I 
well 



c □ 

C J 

c □ 

c □ 



c □ 



c □ 



■ j 
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±5-. P!sbs.b list the 2 noal 'sw^rclrg aspflcte of your Job: 



1 . 



2 . 



46. pimbs list the 2 rriDBt b g n r ;c nt civaiienpBB conrrcri ig school ;ol r ss o r 3 

■L 



2 . 



47. It you would like His results of our surrey, please enter your emal I address: 



-2 Additional Commants: 



Your perspective is important to usE 
Thar k you for participating in our study! 

Please return your survey in the enclosed envelope by 

MAY 20, 2005 

Career Connections Research Center 
1903 W« Michigan Avenue 
Western Michigan University 
3302B Sangren Hall 
Kalamazoo. Ml 49006-5256 
P lio ne : 269-387-61 31 Fax : 2 69 -337-2826 



14 
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APPENDIX E 
WEB SURVEY 



C i-ilc-Ie-:: 



Career Connections Research Center 
Western Michigan University 
269.387.6181 

Welcome to the School Counselor 

Survey! 
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SECTION I 

School Counselor Work Setting and 

Caseload 



L PhKH >JI J«ll lurt. 

1 

2 l>: ■ r l 3. M'k hi n r^hefll oz-umilcr ‘.'li ipi Mlh ilu^rii In wiy u i :+ j i P Ihmigh I XT 
r Vn 

r ^ 

CuvlnjA 

Fjaibi contact ;lrrja;--.8''A-i:> :i *±. if yva him izy cv.-acucc;- laEjrdnif dis jun'sj. 
bfa i-^ray: Farand b^p: kBEHPgl 



4^ Complete 

3. Are you assigned to work with students in one specific school or more than one school? 
One sohool 
Mere than one school 

Continue 

Flease contact shl r ev swIhtLi .■• rr dt.edJ lfvou have any qjesctsns regarding tils surrey. 
Online Surveys Powered 3y /sVsv OurcevZ os m 
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HB4 CcmplaLe 



Percent or won thie 

1 ■ 25^ 2E ■ 53^ E-1 - 75% 75 - 1 DO% 



School name (TMS Mil scroll Irslce ot 
wlrdcwj 



r r r r 

cere | | 

r r r r 

tf you *crk In mors than on* actual, please ansMsrtne rema n ng questions oaaec onliie ashool si 
winch you spend Iho moat time. If you opt id ecial litre ad tl tps schools, ploK one or -vltlci to case 
your responses and enter t calc* 



The following questions refer to your "caseload" of students. By 
caseload, we mean the students for whom you are assigned specific 

responsibility. 



5. Dobs your caseload cone let or al I students [In your school or a specific group or groups dT students? 
;Chsck only one) 

■T H\ slucents n Ena school 

r spenJfc g na up (a ) or students 



Corri^e | 

^ease eo triad Enllsy.s/in.vf-.vncr s-tL it you have any xestcos recsTlrg Ms survey 



□nine Surveys Powered By '.v-v.- dj'vcvz tor 
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11% Complete 



6. You indicated that your caseload is comprised of specific groups of students. Please check all 
group definitions that apply. 

I - Specific grade(s) 

I - Alphabetic group 

I - Students in a specific curriculum 

V Special populations 

f~ Other 

Continue | 

Please contact shirlev.swift@wmich edu if you have any questions regarding this survey. 

Online Surveys Powered Bv www SurveyZ com 



12% Complete 

7. Please describe your caseload parameters. 

${piping_text} 



Continue 

Please contact shirlev.swift@wmich.edu if vou have any questions regarding this survey. 



Online Surveys Powered By www.SurvevZ.com 
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WMCtiip lie 



D Vh ■ .h ! -l*£t ai jwur llnv 'J :■ yc 'J ap«nd t»! wen. -y dlipdty wH- s-'.-iIem! ■ ind 1 L 1 on ifl tin 

■j 1 1 1 : i I vi a it m dr 1 , v lit:- witaii • y: 'J hie 1 i i:-\ ¥iz-'hlii-; : i-azlhr will ■ IlpJetiIb^ 

h lI lit i * i 4s n ► li i j iSmiS- u-IIIi ikjup’li 



h .1 1 1 'i no I ratJ’a li ail p will 

2 l ivi imr r ilucg-yi .H ■ c- on y:ui I i-vh mriaacl Hit ictmid ve-.h i V 1 14 TIT NLh'L . 



I*. Gllln n!. an j^ui lefcdl mriadd Sfili ie-w -pm, hnn nwv nc*h« ipirnd e‘Ju-j.-1Ip- 
m rt fm b" 3 Phm fl-'ja 1 l-a M J F. 1 L- L -:l ipin Hi -adiiEilksn ihJia-VH briuw. 

Ii 



II. Rhi* diRk hwn dial ybu nml -drvfdudlty wHh k-.-'Jciii* anya.' 'jd?-3 pm yni. 



r 5 f|; Ldl kl.-Us* I w III a 0 lit rr 

OnH 

■ft- 2 3 the 3-4lirn lim HA 



I fZ+JM Adiif w til n if. ■ -idL> P.h 

hwl 

C • 2 Uiifli 3 -4 Inn lini HA 



r r r r 



r r r r 



I Z. Duteid Hil* ec>qoi pvir, plim Is: Mir. III ■ r :+j ! v \ -:i IhiIr un Mhidi >□. 

inn 




f :#'il tip rr ■: !■ ■. 



Pp4hmt 

i mlBidki i :»■■■ m W- ■: -Efull TU ‘■■wirs 'i-iiJ-’i 



-y In uvn 
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21% Conrpete 



Section II • Career Assessment Activities 

The questions in this section will help us understand more about 
how formal career assessments are implemented in your school and 
school counselor involvement in implementing these assessments. 

By formal career assessments, we mean assessments or 
assessment batteries or "packages" designed to provide information 
regarding students' career interests, aptitudes, abilities, or other 
constructs that help students make career choices and/or choose 

career paths. 



13. Generally, do the students on your caseload participate in any career assessment(s) at some time 
during their secondary education ? 

C Yes 

C No 

Continue 

F ease contact sm r ev swtr.ffwT :r .e]j r you have any questions regaraing tnis survey. 

Online Surveys sowe^ed 5y www SurvevZ com 



22% Complete 

14. Which students on your caseload typically participate in career assessment(s) during their 
secondary education? (Check only one) 

C All students 

C Only specific groups of students (e g., those in specific career tracks and/or population groups) 

Continue 

Please contact shirlev.swift@wmich.edu if you have any questions regarding this survey. 
Online Surveys Powered By www.SurvevZ.com 
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!:e^C0-rfi5J» 



if. rc-L malted .1**: only *pt:'lc £ r trL?i of «:uavMi on vouoa^los: sarlc-gal* *i :are*' 
•JH9SVT6'rtf*|. PlraePHfllCSSB WfilcPl group. [Ch«* JHtn&l app-lyi 

r GaMp pf*f> 

f~ Career ais :ecTi « i GTE ; of Mi pnes bziz^m 

I" s: jOK» <no iptoi* mm 

r S-jCWO rt!!0 3ft Mffmitll) aU3>3T9^ 

“ SUPtrfi **10 W€ *-3de*Tl|C9qi' CJ M'3V3n3jt'l 
I” SaiWB * -I irr,!Xfl "r.jrin pnMtrcy 

f~ 0™r f 



Questions 16 through 24 focus on students with end without 
disabilities on your caseload, If your caseload does not include a 
specific group of students, please indicate NA. 

Students with disabilities are students who receive special education 

services. 



I( T fetters:-, whs? perceolege of tt* FlutHrl* on ywii tral*;; p.irlc pel* hi wr**f <-93999 m»n(i bV 
■^»I t 4 dim r-g Ch*f **C0rt39JY «v*‘tJ<»t^ 



StdeilE wflh diMb ltte 

t - 3( ■ ii ■ 7ft- 

tw* 33% K% 75% iK% Ha 



SiKeiU ArttfioirtdSJbt** 

1- »- ftl- 7ft- 

few 3ft% S$% 7#% 130% MA 



1 1 Ho'A cHw do*» (he typtoa* etudanfi sit your era Md pe^:o.ip4l* in wm? 3h»BS»irf nl ftffrifl 
i*<oii^fy tOttHOttV? 

BitK^riB *i:h, ct-acltf *1, SftKwnte wthoul csaeiltfH 

MHtTUfi 

prut. Pdtnqd Jkwuiityaf '-v( ol- r^tc Airnumy V 

■Dice a.’ifuirv rar* ore* annually ws-irten 



r r r 



f f r 
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II- Typlc-Uty, sA nT'Jt grit* da- tra dSikHihs wi yM cjmmsc mi partdpaw In cmr in«snwKl 



SiLOtrs win (ttubi!ilj«s 

t l 9 » :g ti 12 



r r r r r r r 



Off* 



r 



woiout OHaah :ea 



s f i a to »i 12 off*? 



r r r r r r r r 



ilysuflrt^cnri "gltMV" fwrfuffwli #Wi o* lffltml tf *s$ tile-*. p*3i* indfcp?-? otnwrgr?d»!»wr. 

_ 4 „ ___ f esse KiK“y grae fTesi wii kiqi m-w x 

3 * 44 * ehtt* gioy) 



4 tddmt» #n* fttnMM* 

r 

S-! jdm:i wlthOvl 

r 




1 1. Whin da dip -- - _ >J ■ 1 1 >■ sn >□- ■ cmlwd Ini puilklpuii mew-ipr -!■:■■]■:■ iiciiIV 



Lluhilg wtlh d iah- Iru 
Ainiw Yl>. hjiAhi 

ointa 1 h.1rjcI h.Ii-jl- 1 K4 illai 



■Juitita h Mil . I j j^i'l I n 
Aiivx Tiu m nil fi i 

ia~tai i:huil h.1mcI NA 



r r r r r 



r r err 



II you mnwiil 'nihil" for iLok-If wfch or wtl^oul -jIi:h:I Mil. pimp Indkrfn ■= 1 1 1 = r p ,h:s bilnw. 

. . s . OI~EI L^LI I‘"|L I *ll [id 1 1 Ll|/lta rtl I 

mill mki i-Tw idm| 



Sludbrla wHh d Bib IIIei 

r 

9ludb-lB Hil:~yu: dNibl lllii 

r 
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24. In your Echoal, wo hm prlmiy rasp or mb! tty tar n , :l<n , i".ii; :h'et iMn^E^I|i] lai llx 
Ttujtili an jQui'CiiHlQid? {Chid -= 1 1 •/ -:ii: 



Ll.itrli v.ill, JtailiiJiiii 



Afhita iv 1 1 ■ _ I . I n 



Jl> 



ULlEI (JP 3PJII 



I ia 



L>Jhi jc sr: 



r 



r 



r 



r 



II ynj unrHiWid "alhEi pci9> sn~ for >■ .'J:ii!i w III <dlis4bl: IIep. i- s +e s wiiwv c- InlaiMngi qumltai. 
21. Whs hdfl p 1 1 ri n> inpanlliliY for linphTiinllng :h ir ■ mninnml|i| lm Ihi il.ft'lr 
■jIs.i.i Nee v^'J' ::rHEk:-.+:"' l^lur. ■. -:ii y -:ii: 

PjrFih i:lmKl :m*i*feiji}i 
L-lIp:lI |iE|’"Lk^1i:a 
^LWjEirk I E kj ' E I j 3 
l|M = hi MlUJl! Ill I E ErJ M E 1 1 3 ; 

Liin 1 hi J !:l'm :lI tjuAm f/Jl Lj I e irj - e i 1 3 
SdRd i-^riEi b I -< h= i 
D El- :l u!" ii sImIit 

El Dim [ 



II yni rnwiil "alhEi pcir hi ' for r'.-iturlE ?? I hail ■: MUIdH ptam vrnmi Itra ■ ■: I k ■* I - u ^iHE'.av 
22. 'Mis Ins p 1 1 ri n> npiparnlbilEy for InpN^iRillr^ :h =k ■ sfiBaienni -I|f i lm 1hi nl.ct'h ¥• ■■«.*. 
■ili.i.i IIeb an j^tn ctmtaudf? CCInc+. wily hh; 

r AikUm e:Ii>:I liu'iiL^) 

1 “ l-ili:il |ji|-j'ilJiiE^s 

r* ii»jBirklBHj-Eiia 

r 1, l|H=h i dj-.n I hi Imr*«|i!> 

f 1, L'dIW l! tall II Ldl ;::iljlEkJ Elia 

f“ L-lllllI kl" II fll Hfcl 

f EwuHMwmi 

f DliH r 
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23. What percentage of the students with disabilities on your caseload typically participate in each of 
the following types of assessments? 

None 1-25% 26 - 50% 51-75% 76 - 100*4 



Basic aptitudes 

r 


r 


r 


r 


r 


Career interest 

r 


r 


r 


r 


r 


Personality profile 

r 


r 


r 


r 


r 


Work values 

r 


r 


r 


r 


r 


Work adjustment 

r 


r 


r 


r 


r 


Physical functioning 

r 


r 


r 


r 


r 



None 


1-25% 


26-50% 


51-75% 


76-100% 


Enter other type of assessment ;Text will scro 
"side of window) 


Other 1 

r 


r 


r 


r 


r 


1 


Other 2 
C 


r 


r 


r 


r 


nn 
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24 WhaJl percentage or Hie elude nls *l:rc-dl dlaatillltlBa on ynircasBUied typically po/t slpsle In each 
of IhB following types of aBawsmenta? 



tiore 


1 - 25% 


2E - 53% 


e" 

U! 

>1— 

ift 


75 - 100% 


&aElc apUtudES 

r 


r 


r 


r 


C 


Career mteresl 

r 


r 


r 


r 


c 


Rersonarty prone 

r 


r 


r 


r 


r 


Wert viuas 

r 


r 


r 


r 


r 


'Wort adjjstnent 

r 


r 


r 


r 


r 


PnyBlcal (UicJtlonlrg 

r 


r 


r 


r 


r 



Enter stfier type of assessment (Text w II ecrull 

wcrii 1-25% 2e-5D% 51-75% 75-iDffSi irsida or wirdsft) ' 



Ctfier 1 

r r r r 

COlEf 2 

c c c r 



r 



r 




223 



25. As a school counselor, how are you involved with implementing career assessments with students 
on your caseload? (Check all that apply) 



Students with disabilities St-oems without d sabilites 



Selecting career assessments) 

r 



Administering career assessments) 

r 

nterpreting career assessments) 

r 



m not nvolvec 



r 



r 

r 

r 

r 



26. For students with disabilities on your caseload, please list up to 5 career assessments (right 
column) and the percentage of students assessed with each (left column|. 



% of students with disab ties 
None 1-25% 20-50% 51-75% Over 75% 



Name of assessment (text will scroll insde of 
window; 



Assessment 1 

r r 

Assessment 2 

r r 

Assessment 3 

r r 

Assessment 4 

r r 

Assessment 5 

r r 
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27. For students without disabilities on your caseload, please list up to 5 career assessments (right 
column) and the percentage of students assessed with each (left column). 



% of students without disao ties 
None 1-25% 26-50% 51-75% Over 75% 



Name of assessnent (text will scroll insde of 
window) 



Assessment 1 

r r r r 

Assessment 2 

r r r r 

Assessment 3 

r r r r 

Assessment 4 

c r c c 

Assessment 5 

r r r r 



r 

r 

r 

r 

r 




Section III - Career Planning Activities 



The next questions will help us understand the time you spend and 
kinds of activities you are involved in regarding career planning with 

the students on your caseload. 



Questions 28 through 29 focus on students with and without 
disabilities. If your caseload does not include a specific group of 
students, please indicate NA. 
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2fl. For each cf the following, please indicate the proportion of students on your caseload with whom 
you typically implement or include in the activity. 



StudEnts w th disabilities 

26- 51- 76- 

None 1-25% 50% 75% 100% NA 



Students without d sabilrlies 

20 - 511 - 75 - 

None 1-25% oQ% 75% 1CC% NA 



Administer career assessments) 

r r r c r r 

Provide information regarding results of career 
assessments) 

r r r c r r 

Assist students to identify career gcals 

c r c c r r 

Provide information abcut career areas and/or cptions 

r r r c r r 

Provide information abcut career requirements 

r r r c r r 

Provide assistance to develop cr mcdiiy an 
educational program plan alignec ivith career gcals 

c r r c r r 

Provide information abcut secondary career and 
technical education programs 

r r r c r r 

Provide information abcut pcst-seccndary career and 
technical education programs 

r r r c r r 

Provide information abcut pcst-seccndary educatcnal 
institutions (e.g. colleges and universities; 

r r r r r r 

Teach career and/or employability skills 

r r r c r r 



r r 

r r 

r r 

r r 

r r 

r r 

r r 

r r 

r r 

r r 



r r 

r r 

r r 

r r 

c r 

r r 

r r 

r r 

r r 

r r 



r r 

c r 

c r 

r r 

r r 

r r 

r r 

c r 

r r 

c r 
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25. For each of I is following career p annlng actlvrtiee, please Indicate Ihe context hi *hleti you 
typically inpteTient ttie astlvdy Min the students on yowr casetoail. 



SiLdenls with dlsat lilies 



Students without :IssdI tes 



One-on-Cna GroLp NA Ora-or-Cne Grcup MA 



Administer career sssassnerfti&i 

r r r r 

Pmtde Infoanadon regarding results o' csreer 
assessments; 

r r r r 

Assist si l cents to Hendiy career goals 

r r r r 

Prn.lde IntbfFiaton *out career areas and-oroptors 

r c c r 

Pro - . Ida nttormadon anojL career requirements 

r r r r 

F"ov da dssl5ls.is= la daveiop or fnxsy an 
educadcnal program p ar allgrec with cdteer goals 

r r r r 

Prrnlde hlDnnaflon amutseccndary career and 
team cal Education programs 

r r r r 

Prcr.lde Infcmadon asout pcst-secondary career and 
technical Education programs 

r r r r 

RTff.lda infotnaBon acout post-seocndary erUcattnd 
irsttudonE fa g. colleges and urf.ieisJUes;. 

r r r r 

Teact career ard'oreTplayasllty stlls 

r r r r 



r r 



r r 



r r 



r r 



r r 



r r 



r r 



r r 



r r 



r r 
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30. Please indicate how important it is that students on your caseload experience each of the following. 



St-cents with c sabilites 



Students without disabilities 



Somewhat Somewhat 

Not Important mpcrtarrt Very Important Not mpcrtant mpcrtant Very Important 



Participate in career assessmentis) 

r r r r 

Receive information regarding results of career 
assessments) 

r r r r 

Receive assistance in identifying career goals 

r r r r 

Receive information about career areas and/or options 

r r r r 

Receive information about career requirements 

err r 

Receive assistance to develop cr modify an 
educational program plan aligned with career gcals 

r r r r 

Receive information about secondary career and 
technical education programs 

r r r r 

Receive information about pcst-secendary career and 
technical education programs 

r r r r 

Receive information about pcst-seccndary educational 
institutions (e g. colleges and universities] 

r r r r 



r 



r 

r 

r 

r 



r 



r 



r 



r 



r 



r 

r 

r 

r 



r 



r 



r 



r 
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St-ttiQn |V 



ReCruititUill Jud Support ror Students iA Non U j U ■ Lien Jl Oc^upitiOi^ 

The next questions will help us understand how you might be 
involved in recruiting and/or supporting students in academic or 
career and technical {GTE) education programs considered 
nontraditional (NT) for their gender. 

Nontradition a I (NT) programs are generally defined as those in which 
students of one gender represent 25% or less of those enrolled in the 
program [e.g^ females in computer programming, males in nursing). 



IT. In general, to who: extent is N'T ’enrollment in curricula of pro-grama emphasized in you* school? 
I C he : k only one} 

r 4 Not a; f* Scmew*’^ ^ Sigitifeafitiy 



For questions 32 - 3$, please indio-aie (a|> Wether you do the activity ana [h] whether you consider the 

activity to be 'Within the scope of your job. 

32. Review guidance materials and practices for... 

Do the Mtivdy fltlrvity is within (he ScPpe df ydur job 

y« No r« ftto 



Gender bias ond or "representation' 

r r r 

JUtiJi! or ethnic bias and-'or 'representation" 

r r c 



r 

r 
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ill. navtawieuinDJluTi and Ettiir -tetrad mtltruli ter.. 



l>c lira i:lhlli 


Azhs\ N ii w Hi - 13ra mcapm ■-! ^air \at 


Yma Sfe? 


f n P+: 


& i n re 1 Hh urdlor 'n^mnfcilR'ir 

r r 

HkIiI ar iltinr LIji uidkr 'itpreipnlirtiEin" 


r r 


r r 


r r 



14. Frnvkp f :-c j - ' r. lifctmiUcn rfb-z*jl rrarriiadHranil Eimn undkr pnyiwni In.. 



Dc-lhn h = 1 1 ■> 1 : r 


JV: 1 1 *1 1 |r h» Vi i 1 1 i ii ttip n -j :■ pe t 1 ^ir 


'■'*i ha 


‘-'■a Ha 


Sludt-lB 

r r 


r r 


PinntilgurJm 

r r 

EdLEilan 


r r 


r r 


r r 


r r 


r r 


11. i-aHE 'jL-iicii: -Eijdrcl-i. . 

□ a Hit Ktl hi ly 


*a:! v“.v ii w -.-I” ll.c «:':|.'E lH ^nuijeta 


Vn Hu 


111 Hi- 


&IHK 1 Jirdinr ilhric b mi 

r r 


r r 


hi cniri indmi pr-sgiHrii 

r r 


r r 


luiriiHninl wnt-ilp" 

r r 


r r 
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se. Actively recm it .. 



Yes 



Do the adlvlty 

l\o 



■i.ctivrt>' is within tha scope oT your job 
Yes No 



Females to co urses and/cr programs In "he STEM 
areas (science, technology, engineering, mathj 

r r r r 

Sludents Tor CTE. programs non-tradlllcnal Tor Hielr 
gender 

r r r r 



37. Provide role models and/or menlori t-cr ... 
Do the activity 

Yes Ns 



Activity Is within the scope sf your Job 
Yea No 



Females enrolled In the STEM areas 

r r r r 

BTudsnte enrollec In CTE programs n ontradlllonal 
for thslr gender 

r r r r 



30. Provide support groups, peer counseling, cr cthe r su sport strategies Tor... 



Do thB activity 


Activity Is within the scope -at your Job 


Yes No 


Yes 


No 


Females enrolled In the STEM areas 






r r 


r 


r 


Bludente enrollec In CTE programs n ontradlllonal 
for their gender 






r r 


r 


r 
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at. Uarrlfy .. 



L*-: lira HEUtllf AeDiHi- Ii wilhn flu scapa at yoir jab 

YU hfcj YU ha 

9TEM ■iiu vtaiN appailiirf'tlHj Isn lh< :anhunlj' la-r 
lur iIe ilu-^rii 

r r r r 

ha nil id It anil mdiIi apparhiitllvt! la Ihra -jl' nruii ly 
lai iluzc--l; 

r r r r 



fc-zn-iii V 

Dc ■■■spre zhie Inhzrmvtaan 

The fallowing questions- will tell us a little about your badkground. In 
studies like these, we like to compare the experiences and 
perspectives of people of varying backgrounds. 



-U. Whil n ywiT u«ndiiT 

r 1 -nrUi ■*“ Mill 

II. Whil n ys-.r ruelal or rlnnra lk-Kl||:gmEl 7 
l:lii> ui ATihzdii.Hnnt.zin 
^ AiTHlziii I'd cii ur ^Jaikui hbihv 
^ H-itan 

hYlfr u- L '-iiui in 
" I ul ru.H i;«i- «. 

hilvrhnnluiD'iiliii ■'mlk burin 
l:i an m i . l ih'z il 
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42. What is your age? (Enter number of years} 




42. PI-4355 indicate haw well you think your formal education (undergraduate & graduate} and 
continuing education (inservice training] prepared you to do these things. 

FORMAL education CONTINUING education 

Very Well Moderate y Well Net at All Very We II Moderately Well Wot at All 



Meet the overall demands cfyourjcb 

r c r r 

Conduct career assessments and planning with 
stucentE with disabilities 

r c r r 

Conduct career assessments anc olanning with 
students without disabilities 

r r r r 

Conduct career assessments anc olanning with 
stucentE I'jith racial cr ethnic backgrounds efferent 
from yours 

r c r c 

Use specie strategieE to recruit stucentE tc curricula 
andi’or programs nonlraditional for their gencer 

r r r r 

Use Epecic strategies tc &upport students enrcilec in 
curricula andcr programs ncntradrticnal for their 
gencer 

r c r r 



r r 



c r 



r c 



r r 



r r 



r r 
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44-. Please list the 2 moot mud n g as cscra c-T your Job: 
First noal rewarding aspect: 




Second most reward ling aspect: 







4 Please list the 2 most a ignlf leant challsng es conrronll ng school counselors 
First challerge: 




Second challenge: 




46. If you would I Ike the results of our survey, pieces enter your emal I s ddreBS. 
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Your perspective is important to usE Thank you for participating in 

our study! 

Com "ug 

=lE3sgcoTl3c1 ailley.Ey.imfrflmicr edj ITvsj have any ^esCons returning ills Btivey 
Onlne Surveys Pawned Ey ■■v-v.- S-nevZ mt 



Thank You 

Thank Your For Participating in Our Survey! 

Your responses have been saved and recorded with ID 636741 

Thank You for Completing this Survey 



Online Surveys Powered By www SurvevZ.com 
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